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The main challenge in using abstractions effectively, is to construct a suitable abstraction for the 
system being verified. One approach that tries to address this problem is that of counterexample 
guided abstraction-refinement (CEGAR), wherein one starts with a coarse abstraction of the 
system, and progressively refines it, based on invalid counterexamples seen in prior model checking 
runs, until either an abstraction proves the correctness of the system or a valid counterexample 
is generated. While CEGAR has been successfully used in verifying non-probabilistic systems 
automatically, CEGAR has not been applied in the context of probabilistic systems. The main 
issues that need to be tackled in order to extend the approach to probabilistic systems is a suitable 
notion of "counterexample", algorithms to generate counterexamples, check their validity, and then 
automatically refine an abstraction based on an invalid counterexample. In this paper, we address 
these issues, and present a CEGAR framework for Markov Decision Processes. 

Categories and Subject Descriptors: D.2.4 [Software Engineering]: Program Verification 

General Terms: Verification 



1. INTRODUCTION 

Abstraction is an important technique to combat state space explosion, wherein a 
smaller, abstract model that conservatively approximates the behaviors of the origi- 
nal (concrete) system is verified/model checked. The main challenge in applying this 
technique in practice, is in constructing such an abstract model. Counterexample 
guided abstraction-refinement (CEGAR) [Clarke et al. 2000] addresses this problem 
by constructing abstractions automatically by starting with a coarse abstraction of 
the system, and progressively refining it, based on invalid counterexamples seen in 
prior model checking runs, until cither an abstraction proves the correctness of the 
system or a valid counterexample is generated. 

While CEGAR has been successfully used in verifying non-probabilistic systems 
automatically, until recently, CEGAR has not been applied in the context of systems 
that have probabilistic transitions. In order to extend this approach to probabilistic 
systems, one needs to identify a family of abstract models, develop a suitable no- 
tion of counterexamples, and design algorithms to produce counterexamples from 
erroneous abstractions, check their validity in the original system, and (if needed) 
automatically refine an abstraction based on an invalid counterexample. In this pa- 
per we address these issues, and develop a CEGAR framework for systems described 
as Markov Decision Processes (MDP). 

Abstractions have been extensively studied in the context of probabilistic systems 
with definitions for good abstractions and specific families of abstractions being 
identified (see Section 6). In this paper, like [Jonsson and Larsen 1991; D'Argenio 
et al. 2001; 2002], we use Markov decision processes to abstract other Markov 
decision processes. The abstraction will be defined by an equivalence relation (of 
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finite index) on the states of the concrete system. The states of the abstract model 
will be the equivalence classes of this relation, and each abstract state will have 
transitions corresponding to the transitions of each of the concrete states in the 
equivalence class. 

Crucial to extending the CEGAR approach to probabilistic systems is to come 
up with an appropriate notion of counterexamples. [Clarke et al. 2002] have iden- 
tified a clear set of metrics by which to evaluate any proposal for counterexamples. 
Counterexamples must satisfy three criteria: (a) counterexamples should serve as 
an "explanation" of why the (abstract) model violates the property (b) must be 
rich enough to explain the violation of a large class of properties, and (c) must be 
simple and specific enough to identify bugs, and be amenable to efficient generation 
and analysis. 

With regards to probabilistic systems there are three compelling proposals for 
counterexamples to consider. The first, originally proposed in [Han and Katoen 
2007a] for DTMCs, is to consider counterexamples to be a multi-set of executions. 
This has been extended to CTMCs [Han and Katoen 2007b], and MDPs [Aljazzar 
and Leue 2007]. The second is to take counterexamples to be MDPs with a tree- 
like graph structure, a notion proposed by [Clarke et al. 2002] for non-probabilistic 
systems and branching-time logics. The third and final notion, suggested in [Chat- 
terjee et al. 2005; Hermanns et al. 2008], is to view general DTMCs (i.e., models 
without non-determinism) as counterexamples. We show that all these proposals 
are expressively inadequate for our purposes. More precisely, we show that there 
are systems and properties that do not admit any counterexamples of the above 
special forms. 

Having demonstrated the absence of counterexamples with special structure, we 
take the notion of counterexamples to simply be "small" MDPs that violate the 
property and are simulated by the abstract model. Formally, a counterexample for 
a system M. and property ips will be a pair {£ , TV), where £ is an MDP violating the 
property ips that is simulated by A4 via the relation 1Z. The simulation relation has 
rarely been thought of as being formally part of the counterexample; requiring this 
addition does not change the asymptotic complexity of counterexample generation, 
since the simulation relation can be computed efficiently [Baier et al. 2000], and 
for the specific context of CEGAR, they are merely simple "injection functions". 
However, as we shall point out, defining counterexamples formally in this manner 
makes the technical development of counterexample guided refinement cleaner (and 
is, in fact, implicitly assumed to be part of the counterexample, in the case of non- 
probabilistic systems) . 

One crucial property that counterexamples must exhibit is that they be amenable 
to efficient generation and analysis [Clarke et al. 2002] . We show that generating the 
smallest counterexample is NP-complcte. Moreover it is unlikely to be efficiently 
approximablc. However, in spite of these negative results, we show that there is a 
very simple polynomial time algorithm that generates a minimal counterexample; 
a minimal counterexample is a pair (£,1Z) such that if any edge/ vertex of £ is 
removed, the resulting MDP no longer violates the property. 

Intuitively, a counterexample is valid if the original system can exhibit the "be- 
havior" captured by the counterexample. For non-probabilistic systems [Clarke 
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et al. 2000; Clarke et al. 2002], a valid counterexamples is not simply one that 
is simulated by the original system, even though simulation is the formal concept 
that expresses the notion of a system exhibiting a behavior. One requires that the 
original system simulate the counterexample, "in the same manner as the abstract 
system" . More precisely, if 1Z is the simulation relation that witnesses £ being sim- 
ulated by the abstract system, then (£,1Z) is valid if the original system simulates 
£ through a simulation relation that is "contained within" 1Z. This is one techni- 
cal reason why we consider the simulation relation to be part of the concept of a 
counterexample. Thus the algorithm for checking validity is the same as the algo- 
rithm for checking simulations between MDPs [Baier et al. 2000; Zhang et al. 2007] 
except that we have to ensure that the witnessing simulation be "contained within 
R" . However, because of the special nature of counterexamples, better bounds on 
the running time of the algorithm can be obtained. 

Finally we outline how the abstraction can be automatically refined. Once again 
the algorithm is a natural generalization of the refinement algorithm in the non- 
probabilistic case, though it is different from the refinement algorithms proposed 
in [Chatterjee et al. 2005; Hermanns et al. 2008]; detailed comparison can be found 
in Section 6. We also state and prove precisely what the refinement algorithm 
achieves. 

1.1 Our Contributions 

We now detail our main technical contributions, roughly in the order in which they 
appear in the paper. 

(1) For MDPs, we identify safety and liveness fragments of PCTL. Our fragment 
is syntactically different than that presented in [Desharnais 1999b; Baier et al. 
2005] for DTMCs. Though the two presentations are semantically the same for 
DTMCs, they behave differently for MDPs. 

(2) We demonstrate the expressive inadequacy of all relevant proposals for coun- 
terexamples for probabilistic systems, thus demonstrating that counterexam- 
ples with special graph structures are unlikely to be rich enough for the safety 
fragment of PCTL. 

(3) We present formal definitions of counterexamples, their validity and consis- 
tency, and the notion of good counterexample-guided refinements. We dis- 
till a precise statement of what the CEGAR-approach achieves in a single 
abstraction-refinement step. Thus, we generalize concepts that have been 
hither-to only defined for "path-like" structures [Clarke et al. 2000; Clarke 
et al. 2002; Han and Katoen 2007a; 2007b; Aljazzar and Leue 2007; Hermanns 
et al. 2008] to general graph- like structures 1 , and for the first time formally 
articulate, what is accomplished in a single abstraction-refinement step. 

(4) We present algorithmic solutions to all the computational problems that arise in 
the CEGAR loop: we give lower bounds as well as upper bounds for counterex- 
ample generation, and algorithms to check validity and to refine an abstraction. 



iEven when a counterexample is not formally a path, as in [Clarke et al. 2002] and [Hermanns 
et al. 2008], it is viewed as a collection of paths and simple cycles, and all concepts are defined 
for the case when the cycles have been unrolled a finite number of times. 
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(5) A sub-logic of our safe-PCTL, which we call weak safety, does indeed admit 
counterexamples that have a tree-like structure. For this case, we present an 
on-the-fiy algorithm to unroll the minimal counterexample that we generate 
and check validity. This algorithm may perform better than the algorithm 
based on checking simulation for some examples in practice. 

Though our primary contributions are to clarify the definitions and concepts needed 
to carry out CEGAR in the context of probabilistic systems, our effort also sheds 
light on implicit assumptions made by the CEGAR approach for non-probabilistic 
systems. 

1.2 Outline of the Paper 

The rest of the paper is organized as follows. We recall some definitions and no- 
tations in Section 2. We also present safety and liveness fragments of PCTL for 
MDP's in Section 2. We discuss various proposals of counterexamples for MDP's in 
Section 3, and also present our definition of counterexamples along with algorithmic 
aspects of counterexample generation. We recall the definition of abstractions based 
on equivalences in Section 4. We present the definitions of validity and consistency 
of abstract counterexamples and good counterexample-guided refinement, as well 
as the algorithms to check validity and refine abstractions in Section 5. Finally, 
related work is discussed in Section 6. 

2. PRELIMINARIES 

The paper assumes familiarity with basic probability theory, discrete time Markov 
chains, Markov decision processes, and the model checking of these models against 
specifications written in PCTL; the background material can be found in [Rut- 
ten et al. 2004]. This section is primarily intended to introduce notation, and to 
introduce and remind the reader of results that the paper will rely on. 

2.1 Relations and Functions. 

We assume that the reader is familiar with the basic definitions of relation and func- 
tions. We will primarily be interested in binary relations. We shall use TZ, S, T, . . . 
to range over relations and /, g, h, . . . to range over functions. We introduce here 
some notations that will be useful. 

Given a set A, we shall denote its power-set by 2 A . For a finite set A, the number 
of elements of A shall be denoted by \A\. 

The identity function on a set A shall be often denoted by id a- Given a function 
/ : A — > B and set A' C A, the restriction of / to A' shall be denoted by J\a'- 

For a binary relation TZ C A x B we shall often write a TZ b to mean (a, b) G 7Z. 
Also, given a G A we shall denote the set {b G B | a TZ b} by TZ(a). Please note TZ is 
uniquely determined by the collection {TZ(a) \ a G A}. A binary relation TZi G Ax B 
is said to be finer than TZ2 C A x B if TZi Q 7^2- The composition of two binary 
relations TZ\ C A x B and TZ2 C B x C, denoted by TZ2 olZ\, is the relation 
{(a, c) I 3b G B. aTZxb and bTZ 2 c} CAxC. 

We say that a binary relation TZ C A x B is total if for all a G A there is a b G B 
such that aTZb. We say that a binary relation TZ C A x B is junctional if for all 
a G A there is at most one b G B such that aTZb. There is a close correspondence 

ACM Transactions on Computational Logic, Vol. V, No. N, Month 20YY. 



5 



between functions and total, functional relations: for any function / : A — > B, the 
relation {(a, /(a)) a 6 A} is a total and functional binary relation. Vice-versa, one 
can construct a unique function from a given total and functional binary relation. 
We shall denote the total and functional relation given by a function / by rel/. 

A preorder on a set A is a binary relation that is reflexive and transitive. An 
equivalence relation on a set A is a preorder which is also symmetric. The equiva- 
lence class of an element a G A with respect to an equivalence relation =, will be 
denoted by [a] = ; when the equivalence relation = is clear from the context we will 
drop the subscript =. 

2.2 DTMC and MDP 

Kripke structures. A Kripke structure over a set of propositions AP, is formally 
a tuple K. = (Q, qx, — L) where Q is a set of states, qx G Q is the initial state, 
— >C Q x Q is the transition function, and L : Q — > 2 AP is a labeling function that 
labels each state with the set of propositions true in it. DTMC and MDP are 
generalizations of Kripke structures where transitions are replaced by probabilistic 
transitions. 

Basic Probability Theory. For (finite or countable) set X with a- field 2 X , 
the collection all sub-probability measures (i.e., where measure of A < 1) will be 
denoted by Prob<i(A). For /j, G Prob<i(A), and A C X, n(A) denotes the measure 
of set A. 

Discrete Time Markov Chains. A discrete time Markov chain (DTMC) over 
a set of propositions AP, is formally a tuple M. = (Q,qx,S, L) where Q is a (finite 
or countable) set of states, qx G Q is the initial state, 5 : Q — > Prob<i(Q) is the 
transition function, and L : Q — > 2 AP is a labeling function that labels each state 
with the set of propositions true in it. A DTMC is said to be finite if the set Q is 
finite. Unless otherwise explicitly stated, DTMC's in this paper will be assumed to 
be finite. 

Markov Decision Processes. A finite Markov decision process (MDP) over a set 
of propositions AP, is formally a tuple M = (Q, qx, 8, L) where Q, qx, L are as in the 
case for finite DTMCs, and 5 maps each state to a finite non-empty collection of 
sub-probability measures. We will sometimes say that there is no transition out of 
q G Q if S(q) consists of exactly one sub-probability measure which assigns to all 
states in Q. For this paper, we shall assume that for every q, q' and every fi G S(q), 
n(q / ) is a rational number. From now on, we will explicitly drop the qualifier "finite" 
for MDP's. In the presence of a scheduler that resolves nondeterministic choices, a 
MDP becomes a (countable) DTMC and a specification is satisfied in an MDP if 
it is satisfied under all schedulers. 

Remark: In the presence of memoryless scheduler S, resulting DTMC is bisimilar 
to a finite DTMC M s which has the same set of states as M, the same initial state 
and the same labeling function, while the transition out of a state q is the one given 
by the memoryless scheduler S. 

Suppose there are at most k nondeterministic choices from any state in M.. For 
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some ordering of the nondeterministic choices out of each states, the labeled underly- 
ing graph of an MDP is the directed graph G = (Q, {Ei}^ =1 ), where (gi, #2) € Ei iff 
/i(<72) > 0, where \i is the ith choice out of q\\ we will denote the labeled underlying 
graph of M by Gi{M). The unlabeled underlying graph will be G' = (Q 7 Li^ =1 Ei) 
and is denoted by G{M). The model checking problem for MDPs and PCTL is 
known to be in polynomial time [Bianco and dc Alfaro 1995]. The following notation 
will be useful. 

Notation: Given an MDP M = (Q,gx,<5, L), a state q G Q and a transition ^ G 
5(q), we say that post(/i, q) = {q' e Q /i(g') > 0.} 

Unrolling of a MDP. Given a Ai = (Q,qj,S, L), natural number k > 0, and 
g G Q we shall define an MDP M q k = (Q 9 k , (g, k), <5)?, L^) obtained by unrolling the 
underlying labeled graph of M up-to depth k. Formally, M.\ = (Q 9 k , (g, k), <5)?, L^), 
the k-th unrolling of M. rooted at q is defined by induction as follows. 

-Ql = {(q, k)} U (Q x {j G N I < 3 < k}). 
-For all (q',j) &QI L((q',j)) = L(q') . 

—For all (q',j) G Q^, S((q',j)) = | [i G S(q')} where [i j is defined as- 

(1) fi°(q") = for all q" G Q^, and 

(2) for < j < k, ^ +1 (q") = fi(q') if q" = {q',j) for some q 1 G Q and 
otherwise. 

Please note that the underlying unlabeled graph of A4 q k is (directed) acyclic. 

Direct Sum of MDP's. Given an MDP's M = (Q, q x , 5, L) &ndM' = (Q', q' T , 5', V) 
over the set of propositions AP, let Q + Q' = Q x {0} UQ'x {1} be the disjoint sum 
of Q'. Now, define 5 + 5' : Q + Q' -> Prob<i(Q + Q') and L + L' : Q + Q' AP as 
follows. For all g G Q and G Q', 

-(5 + 0)) = x {0} I M G and (<5 + 6>)((q', 1)) = {/x' x {1} | G 6>(q')} 

where fj, x {0} and /x' x {1} are defined as follows. 

— n x {0}(<7i,0) = fj,(qi) and fi x {0}(qi,l) = for all q 1 G Qi and q[ G Q 2 . 
— // x {1}(<7i, 0) = and x {l}(q[, 1) = //(gi) for all 9l G Qi and q[ G Q 2 . 
(L + !_')(«, 0) = L(g) and (L + L')(g', 1) = L'(g'). 

Now given q G Q + Q', the MDP (X + _M') 9 = (Q + Q', q,S + S',L+ L') is said to 
be the direct sum of M and A4' with q as the initial state. 

Remark: MDP's M = (Q, qi, 5, L) and M' = (Q', g^, (5', L') are said to be disjoint 
if Q n Q' = 0. If MDP's M = (Q, q x , 5, L) and M' = (Q', q' x , 5', L') are disjoint, then 
Q + Q' can be taken to be the union QuQ'. In such cases, we will confuse (q, 0) 
with q, (q', 1) with q' , fi x {0} with /z and // x {1} with ^' (with the understanding 
that fi G S(q) takes value on any q' G Q' and \i G 5(g') takes value on any 

qeQ). 

2.3 Simulation 

Given a binary relation 1Z on the set of states Q , a set A C Q, is said to be 1Z- 
closed if the set 7^(A) = {t \ 3q G A, qlZt} is the same as A. For two sub-probability 
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measures /x, fx' G Prob<i(Q), we say \J simulates /i wii/i respect to a preorder 7?. 
(denoted as /i -<n fi') iff for every 7£-closed set A, /x(A) < ti'(A). For an MDP 
.M = (Q, qx, 5, L), a preorder 7£ on Q is said to be a simulation relation if for every 
q 1Z q', we have that L(q) = L(g') and for every [i G (5(g) there is a // G £(g') such 
that /x //. 2 We say that g ^ g' if there is a simulation relation 7Z such that 
qlZq'. 

Given an equivalence relation = on the set of states Q, and two sub-probability 
measures /U, n' G Prob<i(Q) we say that [i is equivalent to // with respect to = 
(denoted as /i //) iff for every =-closed set A, /i(A) = m'(A). For an MDP 
A1 = (Q,qx,S, L), an equivalence = on Q is said to be a bisimulation if for every 
g 1Z q', we have that L(g) = L(g') and for every [i G (5(g) there is a // G (5(g') such 
that ^ «= We say that g « q' if there is a bisimulation relation = such that 
g = g'. 

Remark: The ordering on probability measures used in the definition of simulation 
presented in [Jonsson and Larsen 1991; Segala and Lynch 1994; Baier et al. 2005] 
is based on weight functions. However, the definition presented here, was originally 
proposed in [Desharnais 1999a] and shown to be equivalent [Desharnais 1999a; 
Segala 2006]. 

We say that MDP M = (Q, qx, S, L) is simulated by M' = (Q', q' x , 5' , L') (denoted 
by M. if! A4') if there is a simulation relation 1Z on the direct sum of M. and M.' 
(with any initial state) such that (gi,0) 1Z (q T ,l). The MDP M is said to be 
bisimilar to M' (denoted by M ~ M') if there is a bisimulation relation = on the 
direct sum of M. and M! (with any initial state) such that (gi,0) = (q' Xl 1). 

As an example of simulations, we have that every MDP M = (Q, gj, 5, L) sim- 
ulates its fc-th unrolling. Furthermore, we also have that if k < k' then the fc'-th 
unrolling simulates the fc-th unrolling. 

Proposition 2.1. Given an MDP M. with initial state qx and natural numbers 
k, k' > such that k < k! . Let M 9 k x and be the k-th and k' -unrolling of M. 

rooted at qx respectively. Then M q ^ ■< M and M q ^ ■< M q ^ . 

Simulation between disjoint MDP's. We shall be especially interested in sim- 
ulation between disjoint MDP's (in which case we can just take the union of state 
spaces of the MDP's as the state space of the direct sum). The simulations will also 
take a certain form which we shall call canonical form for our purposes. In order 
to define this precisely, recall that for any set A, idA is the identity function on A 
and that rel^ is the relation {(a, a) | a G A}. 

Definition: Given disjoint MDP's M = (Q,gi,<5, L) and M' = (Q\q' x ,S', L'), we 
say that a simulation relation 1Z C (Q + Q') x (Q + Q') on the direct sum of 
Q and Q' is in canonical form if there exists a relation TZ\ C Q x Q' such that 
1Z = re\ idQ U TZi U rel idq , . 



2 It is possible to require only that L(q) C L(g') instead of L(q) = L(g') in the definition of 
simulation. The results and proofs of the paper could be easily adapted for this definition. One 
has to modify the definition of safety and liveness fragments of PCTL appropriately. 
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The following proposition states that any simulation contains a largest canoni- 
cal simulation and hence canonical simulations are sufficient for reasoning about 
simulation between disjoint MDP's. 

Proposition 2.2. Given disjoint MDP's M = (Q, q x , 6, L) and M' = (Q', q x , 5', L'), 
let TZ C (Q + Q') x (Q + Q') be a simulation relation on the direct sum of Q and Q'. 
Let TZ\ = TZf) (Q x Q'). Then the relation TZo = rel,d Q UT^i U reljd Q , is a simulation 
relation. 

Proof. Clearly TZo is reflexive and transitive. Fix q G Q and q' G Q' such that 
q TZo q' ■ Please note that by definition qTZq'. Hence, L(q) = L(q'). We need to 
show that for any [i G S(q) there is a /ii G S'(q') such that [i <u a f-i- Since 1Z is a 
simulation relation there is a a [i! G 5'(g') such that [i <n [j! . Fix one such fi' . We 
claim that p <n a // also. 

We need to show that for any 7£o-closed set Qo C Q U Q', we have that /x(Qo) < 
/z'(Qo). Now let Qi = Qo n Q and Q 2 = Q n Q'. We have that /z(Q ) = /u(Qi) and 
//(Qo) = //(Q2). Thus, we need to show that /t(Qi) < /*'(Q2). 

Now, consider the set 7£(Qi) = {(ft G Q U Q' | 3</ a G Qi s.t. q a 7Z qb}. Now since 
1Z is a preorder, 7£(Qi) is 7?.-closed and Qi C TZ(Qi). From Qi C 7£(Qi), we can 
conclude that /i(Qi) < /i(7£(Qi)). Also, since 7£(Qi) is 7?.-closed and /x // we 
have that /i(ft(Qi)) < /t'(7e(Qi)). Hence, we get that /z(Qi) < //(7e(Qi)). Now, 
please note that //(7£(Qi)) = /t'(7£(Qi) n Q'). Hence, the result will follow if we 
can show that ft(Qi) nQ'C Q 2 . 

Pick (jh G 7£(Qi) flQ'. We have by definition that qb G Q' and there exists q a G Qi 
such that q a 1Z qb- Now, please note that as Qi C Q, we get q a TZo qb (by definition 
of T^o )■ Also as Qi C Q , we get that q a G Qo- Since Q is a 7^o-closed set, qb G Qo- 
As qb G Q', we get qb G Q2 also. Since qb was an arbitrary clement of TZ(Qi) H Q', 
we can conclude that TZ(Qi) (1 Q' C Q 2 . □ 

Notation: In order to avoid clutter, we shall often denote a simulation relid Q UTZi U 
re\id Q , in the canonical form by just TZi as in the following proposition. Further, 
if TZ C Q x Q' is a canonical simulation, then we say that any set A C Q U Q' is 
7^-closed iff it is re\ idQ U TZ U relj dq , -closed. 

Proposition 2.3. Given pair-wise disjoint MDP's M — (Qo, <?o, c>o, L ); -Mi = 
(Qi,gi,(5i, Li) and M 2 = (Q2,<72,<$2, U), if Hoi C Q x Qi and TZi 2 C Q x x Q 2 are 
canonical simulations then the relation TZ02 — TZ\2 o 7?-oi C Q x Q2 is a canonical 
simulation. 

2.4 PCTL-safety and PCTL-liveness. 

We define a fragment of PCTL which we call the safety fragment. The safety 
fragment of PCTL (over a set of propositions AP) is defined in conjunction with 
the liveness fragment as follows. 

i/js :=tt ff p (-P) D W-sa^s) D (fcvii s ) n £V(xVl) n r< P (i>L u i> L ) 

i> L : = tt n ff d ^ d (-p) d (*law n ^v?i L ) n (-tv(xvl)) d (^(^ u ^)) 

where P G AP, p G [0, 1] is a rational number and < G {<, <}. Given a MDP M 
and a state q of A4, we say q \\~m ^ if q satisfies the formula tp. We shall drop M 
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when clear from the context. We shall say that M Ih ip if the initial state of M 
satisfies the formula. 

Note that for any safety formula ips there exists a liveness formula ipL such that 
for state q of a MDP M, q I Km V's iff q V'l- Restricting < to be < in the above 
grammar, yields the strict liveness and weak safety fragments of PCTL. Finally 
recall that Otp is an abbreviation for tt U ip. 

There is a close correspondence between simulation and the liveness and safety 
fragments of PCTL — simulation preserves liveness and reflects safety. 

LEMMA 2.4. Let M = (Q,qx,S, L) be an MDP. For any states q,q' £ Q, q ^ q' 
implies that for every liveness formula ipL, if q \\~m ipL then q' \\~m ipL and that for 
every safety formula ips, if <?' 'Km ips then q \\~m V'S- 

Proof. The proof is by induction on the length of the safety and liveness for- 
mulas. We discuss the case when ips is of the form Vv <p (ipL 1 U ipL 2 )- Assume that 
q' \\~m 4>S- We need to show that q \\~m 4>s- Now if q \f m ipL-i then (by induction) 
q' \Ym ipLi- There are two possibilities to consider. If q' ipL 2 then p must be 
1 and < must be <, and so trivially q I Km Pr<i(V'i, 1 U ipL 2 )- On the other hand, if 
q' ¥m ">Pl 2 then by induction q \f M ip L2 and so q \\- M Pr <p (ip Ll U ipL 2 )- 

Now let us consider the case when q Ihyn ipL 1 - Let K C Q x Q be a simulation 
relation such that q TZ q'. Now, let Q C Q be the set {q £ Q | q \\~m V'Li V ipL 2 }• 
Clearly q, q 1 £ Qo. Let So be the restriction of S on Qo- That is 5o(q) = {/Uiq | t 1 £ 
S(q)}. Pick a new label P$ L and for each each q £ Qo let L (<zo) = {Pif> L2 } ff 
<Zo I Km tPl 2 and otherwise. Consider the MDP Mo = (Qo,q, ^o, L ). It is easy to 
see that for any q Q £ Qo, qo I Km ips iff 9o IKm -Pi/>i, 2 )■ 

Let TZo be the restriction of TZ to Q , i.e., TZo = TZ n (Qo x Qo). We first show 
that TZo is a simulation relation on Mo because of the following observations. 

(1) Reflexivity and transitivity of 7Z follows from reflexivity and transitivity of TZ. 

(2) We claim that if A C TZo is TvLo-closed then A must also be 7?.-closed. The proof 
is by contradiction. Assume that there is a q\ £ Q \ Q such that q\ £ 1Z(A). 
Now, pick q £ A such that q TZ q\. By construction, either q \\~m V'ii or 
<Zo I Km "4>l 2 - Since TZ is a simulation, we get by induction that qi Ih^ ip Ll or 
qi 'Km ?Pl 2 - This contradicts q\ Qo- 

From the above claim it is easy to see that if fi <n [i' then ^|q <n a A*'q q - 
Now, let qo TZo q'o an( ^ P^k Mo £ #o (<?())• We have by definition that qo TZq' and 
there is a /i £ S(qo) such that ^\q q = fJ-o- Since TZ is a simulation there is a 
[i' £ S(q' ) such that \i -<u t 1 ' ■ We get by the above observation, ^'q q £ So(q ) 
and no <n m| Qo - 

(3) Similarly we can show that if go T^o q'o then L (qo) = \-o(q' )- 

We have by definition qTZoq 1 . Now, please note we have that q' \\~m P r <ip(^-Pi/>i. 2 )• 
Since OP^ L is a simple reachability formula and q TZo q' , results of [Jonsson and 
Larsen 1991] imply that q IKm Pt <p (OP^, L2 ). Hence, we get q \\~m "0S- d 

Remark: 

(1) The fragment presented here is syntactically different than the safety and live- 
ness fragments presented in [Dcsharnais 1999b; Baier et al. 2005] for DTMCs; 
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Fig. 1. DTMC with a large set of counterexample executions 

the two presentations have the same semantics for DTMCs, but behave differ- 
ently for MDPs. As far as we know, the safety fragment of PCTL for MDPs 
has not been discussed previously in the literature. 
(2) Please note that, unlike the case of DTMCs [Desharnais 1999b; Baier et al. 
2005], logical simulation does not characterize simulation for MDP's. One can 
recover the correspondence between logical simulation and simulation, if each 
non-deterministic choice is labeled uniquely and the logic allows one to refer to 
the label of transitions [Desharnais et al. 2000]. 

3. COUNTEREXAMPLES 

What is a counterexample? [Clarke et al. 2002] say that counterexamples must (a) 
serve as an "explanation" of why the (abstract) model violates the property, (b) 
must be rich enough to explain the violation of a large class of properties, and (c) 
must be simple and specific enough to identify bugs, and be amenable to efficient 
generation and analysis. 

In this section, wc discuss three relevant proposals for counterexamples. The first 
one is due to [Han and Katocn 2007a] , who present a notion of counterexamples for 
DTMCs. This has been recently extended to MDP's by [Aljazzar and Leue 2007]. 
The second proposal for counterexamples was suggested in the context of non- 
probabilistic systems and branching time properties [Clarke et al. 2002]. Finally 
the third one has been recently suggested by [Chatterjee et al. 2005; Hermanns 
et al. 2008] for MDPs. We examine the all these proposals in order and identify 
why each one of them is inadequate for our purposes. We then present the definition 
of counterexamples that we consider in this paper. 

3.1 Set of Traces as Counterexamples 

The problem of defining a notion of counterexamples for probabilistic systems was 
first considered in [Han and Katocn 2007a]. Han and Katoen present a notion 
of counterexamples for DTMCs and define a counterexample to be a finite set of 
executions such that the measure of the set is greater than some threshold (they 
consider weak-safety formulas only). The problem to compute the smallest set of 
executions is intractable, and Han and Katoen present algorithms to generate such a 
set of executions. This definition has recently been extended for MDPs in [Aljazzar 
and Leue 2007]. 

The proposal to consider a set of executions as a counterexample for probabilistic 
systems has a few drawbacks. Consider the DTMC shown in Figure 1, where propo- 
sition P is true only in state q% and q\ is the initial state. Let ips = T'<i-s(OP). 
The Markov chain violates property tps f° r ah values of S > 0. However, one 
can show that the smallest set of counterexamples is large due to the following 
observations. 

— Any execution, starting from q\, reaching q 3 is of the form (qiq2) k qz with measure 
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Fig. 2. DTMC A'fnoiree: No tree-like counterexamples 



(l-e) fc *e. Thus the measure of the set Exec = {(<?i<72)' c <3 , 3 I k < n} is 1 — (1 — e) n , 



— Thus the smallest set of examples that witnesses the violation of ips has at least 



it is unclear whether such a set of executions can serve as a comprehensible ex- 
planation for why the system violates the property ■ips- Further, this DTMC also 
violates the property ips = P<i(OP)- However, there is no finite set of execu- 
tions that witnesses this violation. Such properties arc not considered in [Han and 
Katoen 2007a; Aljazzar and Leue 2007]. 

3.2 Tree-like Counterexamples 

In the context of non-probabilistic systems and branching-time properties, [Clarke 
et al. 2002] suggest that counterexamples should be "tree-like". The reason to 
consider this proposal carefully is because probabilistic logics like PCTL are closely 
related to branching-time logics like CTL. Tree-like counterexamples for a Kripkc 
structure /C and property ip are defined to be a Kripke structure £ such that (a) £ 
violates the property ip, (b) £ is simulated by K,, and (c) the underlying graph of 
£ is tree-like, i.e., (i) every non-trivial maximal strongly connected component is a 
cycle, and (ii) the graph of maximal strongly connected components forms a tree. 
[Clarke et al. 2002] argue that this is the appropriate notion of counterexamples 
because tree-like counterexamples are easy to comprehend. Moreover, they show 
that for any Kripkc structure JC that violates an ACTL* formula <p, there is a tree- 
like counterexample £. 

The notion of tree-like counterexamples can be naturally extended to the case of 
MDPs. Formally, a tree-like counterexample for a MDP M and property ips will 
be a (disjoint) MDP £ such that the unlabeled underlying graph G(£) is tree-like, 
£ violates property ips and is simulated by M. However, surprisingly, unlike the 
case for Kripke structures and ACTL*, the family of tree-like counterexamples is 
not rich enough. 

Example 3.1. Consider the DTMC M no tree shown in Figure 2 with initial state 
qi, proposition P being true only in state q^, and propositions P\, P 2 and P4 being 
true only is states qi, q 2 and (74 respectively. Consider the formula ips = P<i((Pi V 
P 2 V U P). Clearly, the DTMC M notr ee violates ip S - 

We will show that there is no tree-like counterexample for Mnotree and formula 
ips, defined in Example 3.1. We start by showing that there is no tree- like DTMC 
counterexample for Mnotree and ips- 



and the set Exec has size 0(n 2 ). 
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Proposition 3.2. Consider the DTMC M no tree and safety formula ip s defined 
in Example 3.1. If T = (Q, qj,5, L) is a DTMC (disjoint from A4 no tree) such that 
T -< Mnotree and T 1/ ips then T is not tree-like. 

Proof. Assume, by way of contradiction, that T is tree-like. Let Qo = {(71,92,93,94} 
and for each 1 < i < 4, let \i qi denote the transition out of qi in M. no tree- 

For each q e Q, let \i q denote the transition out of q in T. Let 1Z C Q x Q 
be a canonical simulation that witnesses the fact that T -< Ai no tree- We have by 
definition, qjIZqi- Please note that since T violates ips the measure of all paths of 
T starting with qj and satisfying (P x V P 2 V P4) M f is 1. 

As T is tree-like, any non-trivial strongly connected components of G(T) is a 
cycle and G, the graph of the strongly connected components (trivial or non-trivial) 
of G(T) form a tree. Without loss of generality, we can assume that the strongly 
connected component that forms the root of G contains qj (otherwise we can just 
consider the DTMC restricted to the states reachable from qj). 

Hence, we have that every state in Q is reachable from qj with non-zero prob- 
ability. From this and the fact 1Z is a canonical simulation, we can show that for 
any state q e Q there is a q' <E Qo such that qlZq'. Also since each state in Q is 
labeled by a unique proposition, it follows that for each q G Q there is a unique 
q 1 £ Qo such that qTZq' (in other words, TZ is total and functional). 

Now, for each 1 < i < 4, let Qi C Q be the set {q € Q | q 1Z qi.} By the above 
observations, we have that Qi's are pairwise disjoint; Q = Qi U Q2 U Q3 U Q4; and 
for each 1 < i < 4, Qi U {qi} is a "/^.-closed set. Since 1Z is a canonical simulation, 
whenever qTZq', we have /J, q (Qi) < [i q >(qi), for each 1 < i < 4. Moreover, we can, in 
fact, prove the following stronger claim. 

Claim: n q (Qi) = Hq'ili) f° r eacn Q^-l' an d 1 < i < 4. 

Proof of the claim: Consider some 9,9' such that qTZq'. We proceed by con- 
tradiction. Assume that there is some i such n q (Qi) < fi q '(qi). Please note that in 
this case 9' ^ 93 (as n q3 {qi) = 0, VI < i < 4). 

There are several possible cases (depending 9' and qi). We just discuss the case 
when 9' is 94 and i is 2. The other cases are similar. For this case we have that 
^q{Q2) < 1- Also note that for j ^ 2, n q (Qj) < H q4 (qj) = 0. Hence, ^ q (Q) < 1. Now, 
pick two new states q neW2 and q neW3 not occurring in Q U Q . Construct a new tree- 
like DTMC T' extending T as follows. The states of T' are QU{q neW2 , q n ew 3 }- Only 
proposition P 2 is true in q neW2 and only proposition P is true in q neW3 . The labeling 
function for other states remains the same. We extend the probabilistic transition 
[i q by letting fj, q (q n ew 2 ) = 1 — /^(Q) an d H q {q n ew 3 ) = (transition probabilities 
to other states do not get affected). The state q n ew 2 has a probabilistic transition 
Mg„e» 2 such that Vq n ew 2 (<lnew 3 ) = \ a nd M?„ e ™ 2 (9) = for any 9 ^ q neW3 - The 
transition probability from q neW3 to any state is 0. For all other states the transitions 
remain the same. 

Now, please note that there is a path it from 91 to 9 (in T and hence in T 
also) with non-zero "measure" such that P\ V P2 V Pa is true at each point in this 
path. Furthermore, at each point in this path, P is false. Consider the path tt' in 
T 1 obtained by extending tt by q ne w 2 followed by q n ew 3 - Now, by construction ir' 
satisfies (Pi V P 2 V P 4 )U P and the "measure" of this path > (as 1 - ^ 9 (Q) > 0). 
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Now the path w' is not present in T and hence the measure of all paths in T' that 
satisfy (Pi V P2 V P4) U P is strictly greater than the measure of all paths in T that 
satisfy (Pi V P2 V P4) U P. The latter number is 1 and thus the measure of all paths 
in V that satisfy (Pi V P 2 V P 4 ) U P > 1. Impossible! □ (End proof of the claim) 

We proceed with the proof of the main proposition. Let IZi C (QuQo) x (QuQo) 
be the reflexive, symmetric and transitive closure of 1Z (in other words, the smallest 
equivalence that contains TZ). It is easy to see that the equivalence classes of TZ\ 
are exactly Qi U {ft}, 1 < i < 4. From this fact and our claim above, we can show 
that the IZi is a bisimulation. 

Observe now that each element of Q 3 C Q must be a leaf node of G, the graph of 
the strongly connected components of G(T). Using this, one can easily show that 
if T\ is the DTMC obtained from T by restricting the state space to Qi U Q2 U Q4, 
then 7i is tree-like. Let M\ be the DTMC obtained from M no tree by restricting 
the state space to Q \ {93} and Q = (Qi U Q2 U Q4) U (Qo \ {93})- It is easy to see 
that the the equivalence relation IZ2 = TZi H (Q x Q) is also a bisimulation. 

Now, let Gi be the graph of strongly connected components of G(T\). Now, fix 
a strongly connected component of G(7i), say C, that is a leaf node of Gi. Fix 
a state q which is a node of C. Since Gi is tree-like and C is a leaf node, it is 
easy to see that post(^ 9 ,g) can contain at most 1 element. Also, we have that 
q E Qi U Q2 U Q4. Now if q g Qi, we have that q (as a state of %) is bisimilar 
to qi (as a state of Mi). However, this implies that post(/u 9 , q) must be at least 
2 as qi has a non-zero probability of transitioning to 2 states labeled by different 
propositions. Hence q g' Qi. If q G Q 2 , then please note that post(fi q ,q) must 
contain an element in Qi which should also be in C. By the above observation 
this is not possible. Hence q £ Q 2 . Similarly, we can show that q ^ Q4. Hence 
q £ Qi U Q 2 U Q 4 . A contradiction. □ 

We are ready to show that Mnotree has no tree-like counterexamples. 

Lemma 3.3. Consider the DTMC Mnotree and formula tps defined in Exam- 
ple 3.1. There is no tree-like counterexample witnessing the fact that Mnotree vio- 
lates ips- 

Proof. First, since ((Pi VP 2 VP3) UP) is a simple reachability formula, if there 
is a MDP £ < Mnotree which violates ibg, then there is a memoryless scheduler S 
such that £ s violates the property [Bianco and de Alfaro 1995]. Now note that if 
we were to just consider the states of £ reachable from the initial state of then 
£ s is also tree-like. In other words, there is a tree-like DTMC that is simulated 
by Mnotree and which violates the property ips- The result now follows from 
Proposition 3.2. □ 

Tree-like graph structures are not rich enough for PCTL-safety. However, it can 
be shown that if we restrict our attention to weak-safety formulas, then we have 
tree counterexamples. However, such trees can be very big as they depend on the 
actual transition probabilities. 

Theorem 3.4. Ifipws is a weak safety formula and M \f ipws, then there is a 
M' such that G(M') is a tree, M' ^ M, and M' 1/ ipws- 

Proof. The result follows from the following two observations. 
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— If the underlying graph G(Mi) of a MDP Mi is acyclic then there is an MDP 

Mi such that G{Mi) is a tree and M\ « M.2- 
— For any strict livcncss formula ipsL, and a state q £ M ii q lh.A/| ipsL then there 

is a fc such that M q k lh ipsL where Ml is the fc-th unrolling of M rooted at q. 

The latter observation can be proved by a straightforward induction on the structure 
of strict liveness formulas. We consider the case when ipsL is ( _, 'P<p(?AsLi U tpsL 2 ))- 

Now if q II — 'V<p(ipsLi U ipsL 2 ) then note that there is a memoryless scheduler 
S [Bianco and de Alfaro 1995] such that q h^s ~^'P<p(' ! Psl 1 U ipsL 2 )- This implies 
that there is a finite set of finite paths of M s starting from q such that each 
path satisfies ipsLi U ^SLi and the measure of these paths > p [Han and Katoen 
2007a]. Now, these paths can be arranged in a tree T nodes of which are labeled 
by the corresponding state of M. If the state q' labels a leaf node then we have 
q' I Km ^sl 2 ; otherwise q' \\- M ipsL^- 

For any state q' e Q labeling a node in T define maxdepth(<7') = max{depth(t) 
t is a node of T and t is labeled by q'}. If q' labels a node of T such that q' \\~m 
"tpsLt but q' \Ym i>SL 2 then nx k q > such that Ml I lh i>SLi (k q > exists by induction 
hypothesis). If q 1 labels a node of T such that q 1 ipsL 2 but Q 1 ¥m V'SLi then 
fix k q i such that M\ I lh ipsL 2 - If Q 1 labels a node of T such that q' V'SLi 

and q' \\~m V'SLi then fix k q > such that M\ , lh V'SLi and M\ , lh ^5l 2 • Now, let 
k = max{maxdepth(<7') + k q > \ q' labels a node of T.} It can now be shown easily 
that M q k lh Vsl- □ 

3.3 DTMCs as counterexamples 

We now consider the third and final proposal for a notion of counterexamples that is 
relevant for MDPs. [Chatterjee et al. 2005] use the idea of abstraction-refinement to 
synthesize winning strategies for stochastic 2-player games. They abstract the game 
graph, construct winning strategies for the abstracted game, and check the validity 
of those strategies for the original game. They observe that for discounted reward 
objectives and average reward objectives, the winning strategies are memoryless, 
and so "counterexamples" can be thought of as finite-state models without non- 
determinism (which is resolved by the strategies constructed). 

This idea also used in [Hermanns et al. 2008]. They observe that for weak- 
safety formulas of the form V< p {ipi U 1P2) where ipi and ^2 are propositions (or 
boolean combinations of propositions), if an MDP M violates the property then 
there is a memoryless scheduler S such that the DTMC M s violates V< p (tpi Ufa) 
(see [Bianco and de Alfaro 1995]). Therefore, they take the pair {S,M S ) to be the 
counterexample. 

Motivated by these proposals and our evidence of the inadequacies of sets of 
executions and tree-like systems as counterexamples, we ask whether DTMCs (or 
rather purely probabilistic models) could serve as an appropriate notion for coun- 
terexamples of MDPs. We answer this question in the negative. 

PROPOSITION 3.5. There is a MDP M and a safety formula ips such that M 1/ 
ipS but there is no DTMC M! that violates ^5 and is simulated by M. 

Proof. The MDP M will have three states go, Qi, Q2- The transition probability 
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from qi and (72 to any other state is 0. There will be two transitions out of qo, 
A*i and M2, where Mi (go) = 0,m(qi) = §,Mi(#2) = | and ^2(90) = 0,/i 2 (<?i) = 
35^2(92) = §■ For the labeling function, we pick two distinct propositions P\ and 
P 2 and let \-(q ) = 0, L(<Zi) = {-Pi} and L(<7 2 ) = {P 2 }- Consider the safety formula 
ips = P<|(X(Pi A -P 2 )) V P<| (X(-Pi A P 2 )). Now M violates Vs- 

Suppose that M! = (Q' , qx, S' ,L') is a counterexample for and ips- Then 
we must have q x lh -.P < |(X(Pi A -P 2 )) A -.P < |(X(-.P l A P 2 )). Now, if M' is a 
DTMC, 5'(qx) must contain exactly one element \i qii . Also since II — 'P<2 (X(Pi A 
-P 2 )) there must be a state such that Pi e L'(<^),P 2 ^ L'(<7i) and H qx (q[) > 
|. Similarly, there must also be a state g 2 such that P 2 £ L'(g 2 ),P. ^ L'(q 2 ) 
and ^ gi (<7 2 ) > |. Now, clearly ^ q' 2 (as they do not satisfy the same set of 
propositions). However, we have that H qx {Q') > | + | > 1. A contradiction. □ 

3.4 Our Proposal: MDPs as Counterexamples 

Counterexamples for MDPs with respect to safe PCTL formulas cannot have any 
special structure. We showed that there are examples of MDPs and properties that 
do not admit any tree- like counterexample (Section 3.2). We also showed that there 
are examples that do not admit collections of executions, or general DTMCs (i.e., 
models without nondeterminism) as counterexamples (Sections 3.1, 3.3). Therefore 
in our definition, counterexamples will simply be general MDPs. We will further 
require that counterexamples carry a "proof" that they are counterexamples in 
terms of a canonical simulation which witnesses the fact the given MDP simulates 
the counterexample. Although we do not really need to have this simulation in 
the definition for discussing counterexamples (one can always compute a simula- 
tion) , this slight extension will prove handy while discussing counterexample guided 
refinement. Formally, 

Definition: For an MDP M = (Q, qx, 5, L) and safety property ips such that M IK 
ips, a counterexample is pair {8, TV) such that £ = {Qs, qs, 5s, Lg} is an MDP 
disjoint from M., E\f ips and 1Z C Qg x Q is a canonical simulation. 

For the counterexample to be useful we will require that it be "small". Our 
definition of what it means for a counterexample to be "small" will be driven by 
another requirement outlined by in [Clarke et al. 2002], namely, that it should 
efficiently gcncratable. These issues will be considered next. 

3.5 Computing Counterexamples 

Since we want the counterexample to be small, one possibility would be to consider 
the smallest counterexample. The size of a counterexample (£,1Z) can be taken to 
be the sum of sizes of the underlying labeled graph of £, the size of the numbers used 
as probabilities in £ and the cardinality of the set 1Z; the smallest counterexample 
is then the one that has the smallest size. However, it turns out that computing 
the smallest counterexample is a computationally hard problem. This is the formal 
content of our next result. For this section, we assume the standard definition of 
the size of a PCTL formula. 

Notation: Given a safety formula, ips, we denote the size of ips by 
We now formally define the size of the counterexample. 
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Fig. 3. A problem instance of exact 3-cover and the constructed MDP 



Definition: Let M = (Q,qj,S, L) be a MDP. The size of M, denoted as \M\, is 
the sum of the size (vertices+edges) of the labeled underlying graph Ge(Ai) and 
the total size of the numbers Li qe Q{^i(q') | q' G Q,yU G 5(q),[i(q') > 0}. The size of 
a counterexample (£,1Z), denoted as \(£,1Z)\, is the sum of the size of £ and the 
cardinality (number of elements) of the relation 1Z. 

Please note that any MDP M. of size n has a counterexample of size < 2n (just 
take an isomorphic copy of M as the counterexample MDP and take the obvious 
"injection" as the canonical simulation relation). 

Theorem 3.6. Given an MDP M, a safety formula ips such that M \f ips, an d 
a number k < 2\M\, deciding whether there is a counterexample (£,TZ) of size < k 
is HP -complete. 

Proof. The problem is in NP because one can guess a counterexample (£,TZ) 
of size k and check if £ violates ips ■ The hardness result is achieved by a reduction 
from the exact 3-cover problem [Garey and Johnson 1979] which is formally defined 
as follows. 

Given a set X such that |X| = 3q and a collection C of subsets of X such 
that for each C G C, \C\ — 3, is there an exact 3-cover for X. In other 
words, is there a collection of pairwise disjoint sets B C C such that 
X = UbsbB. 

Before, outlining the proof, it is useful to recall what a 3-cover (not necessarily 
exact) for X is: The collection B is said to be an 3-cover, if B C C is a collection 
(not necessarily disjoint) such that X = UbebB. 

Note that without loss of generality we can assume that for each x G X there 
is a C G C such that x € C (if this is not the case, we can simply answer no in 
polynomial time). Note that \B\ = q for an exact cover. Also note that X has 
an exact 3-cover B C C iff there is cover B' C C such that \B'\ < q. (Actually no 
collection B' such that \B'\ < q can cover X, so < is mainly a matter of convenience.) 

The reduction as follows. We first construct an MDP M = (Q, qi, 5, L) as follows. 
For the set of states, we take Q = XUCU{s, t} where s and t are two distinct elements 
not in XUC. The initial state q% is taken to be s. There is one probabilistic transition 
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out of s, /j s , such that /j, s (x) = ^ for each x E X and /i s (g) = for all q E Q \ X. 
From each ieX, = {\i x .c \ x E C,C E C} where assigns probability 1 to 
C and otherwise. For each C, there is one probabilistic transition out of C, He, 
which assigns probability 1 to t and is otherwise. There is no transition out of t. 
Finally, the set of propositions, we will pick a proposition P q for each q E Q and P q 
will be true only in the state q. For the safety formula, we take ips — T'<\(ttlA Pt). 
Clearly M violates ips- The reduction is shown in Figure 3. The result now follows 
from the following claim. 

Claim: X has an exact 3-cover B C C iff there is a counterexample (£,1Z) for M 
and ip s of size < 2(2 + Aq) + 7q + 3q(l + [log3<7~|) + Aq. 

Proof of the claim: 

(=>) Assume that B C C is an exact 3-cover of X. We have \B\ = q. Consider an 
MDP M' which is the same as M except that its states are {q \ q E Q} instead 
of Q. Now delete all states C of M! such that C £ B. Let the resulting MDP be 
called £ and the set of its states be denoted by Qg. Note that the G(£ ) has 2 + Aq 
nodes and 7q edges. Furthermore, from the initial state there is a probabilistic 
transition which assigns probability ^ to each {x \ x E X}. It takes 1 + [log 3(7] bits 
to represent 1+ [log 3q~\ (1 for the numerator and [log 3q~\ for the denominator). For 
each x such that x EX there is a probabilistic transition which assigns probability 
1 to B where B E B is such that x E B. Finally, from each B E B there is 
a probabilistic transition that assigns probability 1 to t. The size of the MDP 
£ is seen to be 2 + Aq + 7q + 3q(l + \log'3q]) + Aq. Now, let 1Z be the relation 
{(QtQ) I <7 S Qs}- Clearly (£,1Z) is a counterexample and one can easily check that 
\S,n\ = 2(2 + Aq) + 7q + 3q(l + [log3g]) + 4g. 

(<=) Assume that there is a counterexample (£,TZ) of size < 2(2 + Aq) + 7q + 
3q(l + \log3q}) + Aq. Thus we have that £ ^ M 1 1Z is a canonical simulation 
and £ violates ips- Now note that since every node of M. is labeled by a unique 
proposition, 1Z is functional. In other words for each state q\ of £ is related to at 
most one state of Q. Observe that since the safety formula tp$ is P<i(^^ Pt), there 
is a memoryless scheduler 5 such that £ s violates ips- Let £ S = (Qg, gg, <5f, Lg). 
For each q x E Qs, let \i qi denote the unique probabilistic transition out of £ s . 

Note that we have qsTZs. Consider the set Qx = post(<7£, [i qe ). Since £ is simulated 
by M; it follows that each element of Qx must be labeled by some proposition P x 
for some x E X. Given x E X, if C Q x is the set of states labeled by P x then we 
must have q1Zx for each q E Q x . We also have that /i qe (Q x ) < ^ and Q Xl (lQ X2 = 
for xi 7^ X2- Now note that the probability of reaching P t from qg is 1 in £ s . Hence 
it must be the case that [i q£ (Qx) = 1 and thus Q x ^ % for any x E X. Therefore 
|Qx| > 3g and the total size of the numbers {n qe (q) | fj, qe (q) > 0, q E Qx} is at least 
|Qx|(l+[log3(?l). 

Now given q E Q x consider post(q, ^ q ). Again as the total probability of reaching 
P t is 1 in £ 5 , post(g, ^i q ) cannot be empty. Furthermore, as each q E Q x is simulated 
by x, it follows that each element q' E post(g, fj, q ) must be labeled by a single Pg E C 
such that x E B. Let Qc = Li qe Q x post(q, ^i q ). By the above observations it follows 
that if we consider the set B = {B E C \ Pb labels a node in Qc} then B covers X. 
Also since every state of Qc is labeled by a single proposition, we get |Qc| > \B\. 
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We can show by similar arguments that for each q & Qc the set post(g, n q ) is non- 
empty and each node of post(g, n q ) must be labeled by P t . Let Q t = U g£ Q c post(g, n q ). 
We have that |Q t | > 1. 

Note that the sets Qx, Qt and Qc are pairwise disjoint and do not contain qg. 
Hence, the labeled underlying graph of £, Ge(£), has at least 1 + |Qx| + |Qe| + |Qt| 
vertices. As 1Z is functional, 1Z also contains at least 1 + |Qx| + |Qc| + |Qt| elements. 
Furthermore, it is easy to see that the underlying graph has at least 2 1 Qx + |Qc 
edges; and the total size of numbers used as probabilities in £ is at least |Qx|(l + 
riog(3f)"|) + |Q X | + | Q c | - Hence the total size of (£, R) is at least 2(1 + |Q X | + |Q C | + 
|Q t |) + 2|Q X | + |Qe| + |Qx|(l + Rog(3t)l) + |Q C |. Since \Q t \ > 1 and |Q X | > 3<z; the 
total size is at least 2(2 + 3g + |Q C |) + 6q + |Q C | + 3q(l + |"log(3t)]) + 3q + \Q C \. 
By hypothesis, the total size is < 2(2 + 4q) + 7q + 3q(l + [log 3q~\ ) + 4q and we get 
that | Qc | < q- But | Qc | > \B\ and hence \B\ < q. Since B is a cover; it follows that 
X must have an exact 3-cover. □ 

Not only is the problem of finding the smallest counterexample NP-hard, it also 
in-approximable. 

Theorem 3.7. Given an MDP M, a safety formula ips andn = \M\ + \ips\ such 
that M. \f tps ■ The smallest counterexample for M and ips cannot be approximated 
in polynomial time to within 0{2 lo ^~ € ") unless NP C DTIME(nP°^ lo s(™)). 

Proof. The in-approximability follows from a reduction of the Directed Network 
Steiner Problem [Dodis and Khanna 1999]. Directed Network Steiner Problem is 
formally defined as follows. 

Given a directed graph G, m pairs {sj, ii}£Li a sub-graph G' = (V, E') 
of G satisfies the Steiner condition if Si has path in G' to U for all i. 
The Directed Steiner network problem asks for a sub-graph G' such that 
G' satisfies the Steiner condition and has the smallest size amongst all 
subgraphs of G which satisfy the Steiner condition. 

It is shown in [Dodis and Khanna 1999] that the smallest sub-graph cannot be 
approximated to within O(2 log where n g is the sum m+ size (vcrticcs+edges) 

of G unless NP C DTIME«°' y los( " s) ). Also note that since e is arbitrary the 
smallest sub-graph cannot be found to within O(2 log £ ( n s lo s(™3))). (Changing n g 

to n g \og(n g ) does not make a difference DTIME(rig 0i? ' los( ' ,l3 ' ) )). 

We now give the reduction. Given a graph G = (V,E), let |V| = n v , \E\ = n e , 
n g = n e +n v . Recall, that the network steiner problem has m pairs (si,ti). Let n s 
be the number of distinct Sj's in (sj, U). In other words n s is the cardinality of the 
set {si | 1 < i < m}. Clearly n s < n g . Please note that for the directed network 
Steiner problem we can assume that n g is 0(n e ). 

We construct a DTMC M with states V U {s}, where s is a new vertex, s is the 
initial state of M and has a probabilistic transition [i s such that /J, s (si) = for 
each 1 < i < m and y(i s (w) — if v £ {s,|l < i < m). Every other state v has a 
transition /j, v such that /j, v (v') = — if (v,v') G E; otherwise fi v (v') — 0. Finally, 
we will have as propositions {P v \ v E V U {s}}, where the proposition P v holds 
at exactly the state v. Since it takes 0(log(n g )) bits to represent the size of 
DTMC M is easily seen to be n v + 1 + n e + n s + (n e + n s )(0(\og(n g ))). 
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Consider the safety formula ips = ipsi V -0S2 where if>Si = VI!li ^<o(^( s i A 
(^<o(Oti))))and^ 2 =V^i^<o(X«i). 

The sum n m = \M \ + \ips\ is easily seen to be 0(n g log(n g )). 

Claim: 

(1) If G has a sub-graph G' = {V , E') with \V{\ = m and \E'\ = n 2 such that G' 
satisfies Stcincr condition then M. has a counterexample of size = 2ni + 2 + 
n 2 + n s + (n 2 + n s )(log(n fl )). 

(2) If the DTMC A4 has a counterexample (£,1Z) such that Gg{£), the underlying 
labeled graph of £, has ni + 1 vertices and n 2 +n s edges then the graph G has a 
sub-graph G' = (V , E') with \V(\ < m and |£"| < n 2 such that G" satisfies the 
Steiner condition. Furthermore, |(£,7£)| > 2n 1 + 2+n 2 +n ;! + (n2+n s )(log(n g )). 

Proof of the claim: 

(1) First assume that G has a sub-graph G' = (V',E') with less than m vertices 
and less than n 2 edges such that G' satisfies the Steiner condition. Consider 
the DTMC M' obtained from M by restricting the set of states to V U {s}. 
Now take an isomorphic copy of M! with {v\v 6 V'} U s as the set of states 
and call it £. Clearly, £ violates ips and the relation {(u, u) \ u € V U s} is a 
canonical simulation of £ by M.. Hence (£,1Z) is a counterexample and it is 
easy to see that \(£, 1Z)\ < 2n 1 + 2 + n 2 + n s + (n 2 + n s )(log(n g )). 

(2) Let £ = (Q £ , q £ , 5 £ , L £ ) and G e (£) = (V, {^}^ =1 ). Note that since each vertex 
of M. is labeled by a unique proposition and 1Z is a canonical simulation, 1Z 
must be total and functional (totality is a consequence of the fact that we 
can remove any nodes of £ that are not reachable from q £ ). In other words 
there is a function g : Q V U {s} such that TZ = re\ g . Again the definition of 
simulation and the construction of DTMC M gives us that if (qi,q 2 ) G Uj =1 Ej, 
we must have (g(qi), g(q 2 )) e£U {(s,Si) \ 1 < i < m} and n' qi (q 2 ) < ^ for 
any probabilistic transition fjf E S £ (q 1 ). From the latter observation, we get 
that \(£,K)\ >2n l + 2 + n 2 +n s + {n 2 + n s ){\og{n g )). 

Consider the equivalence relation qi = q 2 defined on Q as qi = q 2 iff g(qi) = 
g(q 2 ). Let [q] denote the equivalence class of q under =. Let G 2 = {V",E"} 
be the graph such that V" is the set of equivalence classes under the relation 
= and ([gi],[<72]) G E" if (171,92) G Uj =1 E'. Please observe first that G 2 is 
isomorphic to a subgraph of (V U {s},E U {(s, Sj) 1 < i < n s }) with the 
function h([q\) = g(q) witnessing this graph isomorphism. Please note that 
by the fact that Aii violates ipi, it can be easily shown that there is a path 
from Si to U in G 2 . Also since A4i violates ip 2 , G 2 contains edges (s,Si) for 
each 1 < i < n s . We get by the above observations G must contain a subgraph 
G' = {V, E') with paths from s, to U for all i such that \V{\ < ri\ and \E'\ < n 2 . 
(End proof the claim.) 

From the above two observations it easily follows that if G has a Steiner sub- 
graph of minimum size n m ; ni and M. has a counterexample of minimum size n min2 
then ^""" 2 = 0(log(n g )). Now assume that there is a polynomial time algorithm 

to compute the minimal counterexample within a factor of O(2 log ("«»)) then 
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this algorithm produces a counterexample of size k < O(2 logl ^"" , ^)n m in 2 • Thus 
the counterexample size is < O(2 lo s e ("-))0(log(n 9 ))n mini . From the proof of the 
part 2 of the above claim it follows that we can extract from the counterexam- 
ple in polynomial time a Steiner sub-graph of G of size < O(2 log e (" m ))n m i ni . 
Now n m is 0(n g log(n g )) and thus we have achieved an approximation within 
0( -2 lo g 1_£ (« 9 i°sK))). The result now follows. □ 

Remark: A few points about our hardness and inapproximability results are in 
order. 

(1) Please note that we did not take the size of the labeling function into account. 
One can easily modify the proof to take this into account. 

(2) The same reduction also shows a lower bound for the safety fragment of ACTL* 
properties as the reduction does not rely on any important features of quanti- 
tative properties. 

Since finding the smallest counterexamples is computationally hard, we consider 
the problem of finding minimal counterexamples. Intuitively, a minimal counterex- 
ample has the property that removing any edge from the labeled underlying graph 
of the counterexample, results in an MDP that is not a counterexample. In order 
to be able to define this formally, we need the notion of when one MDP is contained 
in the other. 

Definition: We say that an MDP M' = (Q', qi,5', L') is contained in an MDP 
M = {Q,qi,6,L) if Q' C Q, L'(q') = L(q') for all q' G Q', and there is a 1-to-l 
function / : 5' — > 6 with the following property: For each q', q" G Q' and fi' G S'(q'), 
f{n') G 5{q'), cither n'(q") = f{p'){q") or n'{q") = 0. We denote this by M' C M. 

Observe that if M! C M. then M! < M.. We present the definition of minimal 
counterexamples obtained by lexicographic ordering on pairs (£,TZ). 

Definition: For an MDP M and a safety property ips, (£,1?-) is a minimal coun- 
terexample iff 

— (£, 1Z) is a counterexample for M and ips and 

— If (£i,1Zi) is also a counterexample for Ai and ips, then 

— £\ Q £ implies that £ x = £; and 

— if £\= £ then C 7^ 2 implies that V,\ =7^2- 

Though finding the smallest counterexample is NP-complete and is unlikely to 
be efficiently approximable, there is a very simple polynomial time algorithm to 
compute the minimal counterexample. In fact the counterexample computed by 
our algorithm is going to be contained in the original MDP (upto "renaming" of 
states). Before we proceed, we fix some notation for the rest of the paper. 

Notation: Given an MDP M. = (Q, qj, S, L), for each q G Q fix a unique element 
q not occurring in Q. Define an isomorphic MDP A4 = (Q, q~j, 8, L) as follows. 

-Q = {q\ q eQ}. 
— S(q) = {p,\ne S(q)} where 
— for each q G Q, p(q) = n(q)- 
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Initially M curr = M 

For each edge (q,qi) € Ei in Gi(M curr ) 

Let M' be the MDP obtained from M curr by setting fi(qi) = 0, where /t is the 
ith choice out of q, in M curr 

If M' 1/ Vs then M curr = Af' 
od <— End of For loop 

Let £ be the MDP obtained from M curr by removing the set of states from M curr 

which are not reachable in the underlying unlabeled graph of Mcurr 
If Qe is the set of states of £, then let relj n j = {(q, q) \ q 6 Qg} 
return(£, rel| n j) 



Fig. 4. Algorithm for computing the minimal counterexample 

-L(?) = L(«). 

We are ready to give the counterexample generation algorithm. The algorithm 
shown in Figure 4 clearly computes a minimal counterexample contained in the 
original MDP upto "renaming" of states (note that the minimality of relj n j is a 
direct consequence of the fact that every state in £ is reachable from initial state 
and hence must be simulated by some state in A4). Its running time is polynomial 
because model checking problem for MDPs is in P [Bianco and de Alfaro 1995]. 

Theorem 3.8. Given an MDP M and a safety formula ips such that M \f ips, 
the algorithm in Figure 4 computes a minimal counterexample and runs in time 
polynomial in the size of M and %ps ■ 

Please note that for safety properties of the form V< p (i(>sM'4''s) an d ^Kpi^S^^P's) 
where ips,ip's are boolean combinations of propositions, if {£ , re I ; n j ) is the coun- 
terexample generated by Figure 4 then £ must be a DTMC. This is because if £ 
violates such a property then there is a memoryless scheduler S such that £ s vi- 
olates the same property (see [Bianco and de Alfaro 1995]). For such properties; 
model-checking algorithm also computes the memoryless scheduler witnessing the 
violation. Thus for such properties, one could initialize M curr to be M Sl where S\ 
is the memoryless scheduler generated when M. is model-checked for violation of 
the given safety property. 

The counterexample returned by the algorithm in Figure 4, clearly depends on 
the order in which edges of Gi(M) are considered. An important research question 
is to discover heuristics for this ordering, based on the property and A4. 

4. ABSTRACTIONS 

Usually, in counterexample guided abstraction refinement framework, the abstract 
model is defined with the help of an equivalence relation on the states of the sys- 
tem [Clarke et al. 2000]. Informally, the construction for non-probabilistic systems 
proceeds as follows. Given a Kripkc structure K, = (Q, qi, — >, L) and equivalence 
relation = on Q such that L(q) = L(g') for q = q'; the abstract Kripke structure for 
K, and = is defined as the Kripke structure /Ca = (Qa, QA, ~^A, \-a) where 

— Q.4 = {[q]= | q £ Q} is the set of equivalence classes under =, 
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Fig. 5. Kripke structure /C ex 



Fig. 6. Its abstraction /C a t, 



~<IA = [Ql] = , 

-[q]= — m W]= if there is some q\ £ [q]= and q[ G [q'] 
-L A ([qh) = L(q). " " 



such that qi — > q[, and 



Example 4.1. Consider the Kripke /C ex structure given in Figure 5 where go 
is the initial state and the state gn is labeled by proposition P (no other state 
is labeled by any proposition). Consider the equivalence relation = which parti- 
tions the set {go, gi, g2, 93, 94, 95, 96, 97, 9s, 99, 9io, 9n} into the equivalence classes 
{9o,9i,93}, {92}, {94}, {95, 9e}, {97,9s}, {99}, {910} and gn. Then the abstract 
Kripke structure, /C a b for )C ex and = is given by the Kripke structure in Figure 6. 
Here {qo, qi, 53} is the initial state and {911} is labeled by proposition P. 

This construction is generalized for MDP's in [Jonsson and Larsen 1991; Huth 
2005; D'Argenio et al. 2001]. To describe this generalized construction formally 
we first need to lift distributions on a set with an equivalence relation e to a 
distribution on the equivalence classes of = 3 . 

Definition: Given [i e Prob<!(Q) and an equivalence = on Q, the lifting of \i 
(denoted by [yu] = ) to the set of equivalence classes of Q under = is defined as 
\»h{[qh) = M{9' € Q I q' ee g}). 

For an MDP M. = (Q, qi,5, L), we will say a binary relation = is an equivalence 
relation compatible with Ai, if = is an equivalence relation on Q such that L(g) = 
L(g') for all q = q' . The abstract models used in our framework are then formally 
defined as follows. 

Definition: Given a set of propositions AP, let M. = (Q, qx, S, L) be an AP labeled 
MDP. Let = be an equivalence relation compatible with M.. The abstract MDP 



3 It is possible to avoid lifting distributions if one assumes that each transition in the systems 
is uniquely labeled, and has the property that the target sub-probability measure has non-zero 
measure for at most one state. This does not affect the expressive power of the model and is used 
in [Hermanns ct al. 2008] . However the disadvantage is that the abstract model may be larger as 
fewer transitions will be collapsed. 
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for A4 with respect to the equivalence relation = is a MDP M.= = (Q=, q=, S=, L=) 
where 

(1) Q= = {[<?]= k e Q} ■ 

(2) g= - [qxh. 

(3) <5=([g] = ) = {yu | 3g' G [g]= and \i\ G S(q') such that [i = [^i] = }- 

(4) U([ q y) = L(q). 

The elements of Q shall henceforth be called concrete states and the elements Q= 
shall henceforth be called abstract states. The relation rel= CQx Q= defined as 
rel^ = {(q, [q]=) \ q G Q} shall henceforth be called the abstraction relation. The 
relation relX C x Q defined as rell = {([g] = ,g') | [q]= G Q=,q = q'} shall 
henceforth be called concretization relation. 

Remark: The relation rel^ is total and functional and hence represents a function 
a which is often called the abstraction map in literature. Please note that one 
can define the equivalence = via the function a. The relation relX is total (not 
necessarily functional) and hence represents a map into the power-set 2 Q . The 
function 7 : Q= — > 2 Q defined as 7(a) = relX(a) is often called the concretization 
map in literature. 

We conclude this section by making a couple of observations about the construc- 
tion of the abstract MDP. First notice that the abstract MDP Ai= has been defined 
to ensure that it simulates M via the canonical simulation relation re\%. Next, we 
show that we can obtain a "refinement" of the abstract MDP M=, by considering 
the abstraction of M. with respect to another equivalence ~ that is finer than =. 
This is stated next. 

Definition: Let M = (Q, qx, S, L) be an MDP over the set of atomic propositions 
AP. Further let = and ~ be two equivalence relations compatible with M such 
that ~C=. The abstract MDP Ai~ is said to be a refinement of M.=. The relation 
rel^ = C x Q= defined as \q\ = ) \ [q']~ C [q] = } is said to be a refinement 

relation for (M~,M=). 

The following is an immediate consequence of the definition. 

Proposition 4.2. Let ~ and = be two equivalence relations compatible with the 
MDP M. such that ~C=. Recall that the refinement relation for {AA^,M. = ) is 
denoted by relS, = . Then rel^ = is a canonical simulation and relS, = o rel^ = rel=. 

5. COUNTEREXAMPLE GUIDED REFINEMENT 

As described in Section 4, in our framework, an MDP M will be abstracted by 
another MDP A4= defined on the basis of an equivalence relation = on the states 
of M.. Model checking M.= against a safety property ips will either tell us that ips is 
satisfied by M= (in which case, it is also satisfied by M as shown in Lemma 2.4) or it 
is not. If Ai= \]/- ips then M.= can be analyzed to obtain a minimal counterexample 
(£ , re I j n j ) , using the algorithm in Theorem 3.8. The counterexample (£, re I j n j ) must 
be analyzed to decide whether (£, reLj) proves that M. fails to satisfy ips, or the 
counterexample is spurious and the abstraction (or rather the equivalence relation 
=) must be refined to "eliminate" it. In order to carry out these steps, we need 
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Fig. 7. The three counterexamples AT ce xi (shown with short dashed edges), IC ceX2 (shown with 
long dashed edges), and /C C ex 3 (shown with short and long dashed edges) 

to first identify what it means for a counterexample to be valid and consistent for 
A4, describe and analyze an algorithm to check validity, and then demonstrate how 
the abstraction can be refined if the counterexample is spurious. In this section, 
we will outline our proposal to carry out these steps. We will frequently recall 
how these steps are carried out in the non-probabilistic case through a running 
example to convince the reader that our definitions are a natural generalization to 
the probabilistic case. 

5.1 Checking Counterexamples 

Checking if a counterexample proves that the system A4 fails to meet its require- 
ments ips, intuitively, requires one to check if the "behavior" (or behaviors) cap- 
tured by the counterexample are indeed exhibited by the system. The formal con- 
cept that expresses when a systems exhibits certain behaviors is simulation. Thus, 
one could potentially consider defining a valid counterexample to be one that is sim- 
ulated by the MDP A4. However, as we illustrate in this section, the notion of valid 
counterexamples that is used in the context of non-probabilistic systems [Clarke 
et al. 2000; Clarke et al. 2002] is stronger. We, therefore, begin by motivating and 
formally defining when a counterexample is valid and consistent (Section 5.1.1), 
and then present and analyze the algorithm for checking validity (Section 5.1.2). 

5.1.1 Validity and Consistency of Counterexamples. In the context of non-probabilistic 
systems, a valid counterexample is not simply one that is simulated by the original 
system. This is illustrated by the following example; we use this to motivate our 
generalization to probabilistic systems. 

Example 5.1. Recall the Kripke structure /C ex given in Example 4.1 along with 
the abstraction /C a b (these structures are given in Figures 5 and 6, respectively). 
The LTL safety-property n(-*P) is violated by K. 3 b- For such safety properties, 
counterexamples are just paths in /C a b (which of course can be viewed as Kripke 
structures in their own right). The counterexample generation algorithms in [Clarke 
et al. 2000; Clarke et al. 2002] could possibly generate any one of three paths in K, a b 
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shown in Fig 7: /C cexi = {90, 9i, 93} -> {92} -> {99} -> {911}, ^cex 2 = {90, 9i, 93} -» 
{94} -> {910} -> {911} and /C ceX3 = {90,91,93} -» {95,96} -» {97,9s} -» {qu}- Now 
each of the counter-examples /C cexi , /Ccex 2 an d ^cex 3 is simulated by /C ex because /C ex 
has a path, starting from the initial state, having 4 states, where only the fourth 
state satisfies proposition P. However, the algorithm outlined in [Clarke et al. 
2000; Clarke et al. 2002] only considers /C C e Xl to be valid. In order to see this, let 
us recall how the algorithm proceeds. The algorithm starts from the last state of 
the counterexample and proceeds backwards, checking at each point whether any 
of the concrete states corresponding to the abstract state in the counterexample 
can exhibit the counterexample from that point onwards. Thus, /C ceX2 is invalid 
because q does not have a transition to q 4 and /C ceX3 is invalid because none of 
90 , 9i , or <7 3 have a transition to q§ (the only state among q 5 and q% that can exhibit 
{95, 96 } {97,9s} -> {9n})- 

The example above illustrates that to check validity, the algorithm searches for a 
simulation relation, wherein each (abstract) state of the counterexample is mapped 
to one of the concrete states that correspond to it, rather than an arbitrary simu- 
lation relation. Thus the "proof" for the validity of a counterexample in a concrete 
system, must be "contained" in the proof that demonstrates the validity of the 
counterexample in the abstract system. Based on this intuition we formalize the 
notion of when a counterexample is valid and consistent. 

Definition: Let M. be an MDP with set of states Q, and = be an equivalence 
relation that is compatible with M. Let tps be a PCTL-safety formula such that 
■M= V V's an d let (£,lZo) be a counterexample for A4= and tps with set of states 
Qg. We say that the counterexample (£, TZq) is valid and consistent with (A4, =) if 
there is a relation 1Z C x Q such that 

(1) 1Z is a canonical simulation (validity); and 

(2) rel^ 0IZCIZ0 (consistency). 

The relation 1Z is said to be a validating simulation. If no such 1Z exists then {£ , 1Zo) 
is said to be invalid for (Ai,=). 

The above definition provides one technical reason for why it is convenient to view a 
counterexample as not just an MDP but rather as an MDP along with a simulation 
relation; we will see another justification for this when we discuss refinement. 

Remark: When the counterexample (£, reLj) is generated as in Theorem 3.8, Qs C 
{a | a e Q=} and the relation relj n j = {(a, a) | a € Qe}. In this case, please note 
that consistency is equivalent to requiring that 1Z C {(a, q) \ q e rell(a)}. In other 
words, consistency is equivalent to requiring that 1Z C rell o relmj. 

We conclude this section by showing that for minimal counterexamples (£,lZo) 
the containment in the consistency requirement can be taken to be equality. 

Proposition 5.2. Let M. he a MDP, = an equivalence relation compatible with 
A4 and ips be a safety formula such that A4= \f ips- If (£,7^0 ) * s a minimal 
counterexample for M= and ^5 and (£,1Zo) is consistent and valid for (M,=) 
with validating simulation 1Z then rel= o 1Z = IZo- 
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Initially R = {(a, q) \ g £ relX(a)} and R oW = 
while (Raid ^ -R) do 
Rold = R 

For each state a G and each fi £ do 

R = {(b, q)£R\b^a}U {(a, q) e R | 3/u' G 5(g). /! ^ fiold /Li'} 
If 7£(a) = then return ("invalid", a, fi, R a id, R) 
11 a = qg and qx £ R(a) then return ( "invalid" , a, fi, R a id, R) 
od <— end of For loop 
od <— end of while loop 

Return ("valid") 



Fig. 8. Algorithm for checking validity and consistency of counterexamples 

PROOF. We have that TZ is a simulation and rel= o TZ C TZq. Since re\% and TZ 
are simulations, so is rel= o TZ. Also since £ \f tps, we get that (£, re\% o TV) is also 
a counterexample for M= and ^5. Thus, if rel^ oTZ C 7?. , then (f , relj o72.) is not 
minimal. Hence rel^ o TZ = TZq. □ 

5.1.2 Algorithm to check Validity of Counterexamples. We now present the al- 
gorithm to check the validity and consistency of a counterexample. We will assume 
that the counterexample is a minimal one, generated by the algorithm in Theo- 
rem 3.8. Thus the counterexample is of the form (£, reLj), where the set of states 
is a subset of {a | a e Q=} and the relation relj n j is {(a, a) | a £ Qs}. The 
algorithm for counterexample checking is then the standard simulation checking 
algorithm [Baicr et al. 2000] that computes the validating simulation through pro- 
gressive refinement, except that in our case we start with R = {(a, q) \ q £ rell(a)}. 
The algorithm is shown in Figure 8. Please note that for the rest of the paper (and 
in the algorithm) by fi < Rold fi' we mean /x ditd Q uR old vid Q£ fi'- 

We will now show that the algorithm in Figure 8 is correct. We start by showing 
that the algorithm terminates. 

Proposition 5.3. The counterexample checking algorithm shown in Figure 8 
terminates. 

PROOF. Let TZq and TZ" be respectively the relations denoted by the variable 
R at the beginning and the end of the n-th iteration of the while loop. A simple 
inspection of the algorithm tells us that either TZ" = TZq or TV{ C TZq. If TV{ = TZq 
then the while loop terminates (and returns "valid" ) . Otherwise the size of relation 
denoted by variable R decreases by at least one. Hence, if TZ™ is never equal to TZq 
then it must be case that TZ" (a) becomes empty for some n and some a £ Qs and 
then the algorithm terminates. □ 

We now show that if the algorithm returns "valid" then the counterexample 
(£, reli n j) is valid and consistent. 

Proposition 5.4. If the algorithm in Figure 8 returns "valid" then the coun- 
terexample (£, reLj) is valid and consistent for (A4,=). 

Proof. Please note that the algorithm returns "valid" only when the while loop 
terminates. At that point the variables R id and R denote the same relation, which 
we shall call TZ for the rest of the proof. Please note as TZ C {(a, q) \ q £ rell(a)}, 
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re\% o 72 C reli n j. Hence, the result will follow if we can show that 72 is a canonical 
simulation. Consider the last iteration of the while loop. Now, a simple inspection 
of the algorithm says that the variable R does not change its value in this iteration. 
The only way this is possible is if for each a G Qs, /x G 8s (a) and each (a,q) G 72, 
there exists // G 8(q) such that fi -<n \j! ■ Also, qi G TZ(qs)- Thus 72 is a canonical 
simulation. □ 

We now show that the if (Qs, relinj) is valid and consistent, then the algorithm in 
Figure 8 must return "valid" . 

Proposition 5.5. If the counterexample (Qg,reli n j) is valid and consistent with 
(M, =) then the algorithm in Figure 8 returns "valid". 

Proof. Assume that the counterexample (Qs, relinj) is valid and consistent. 
Thus there is a canonical simulation 72 C Q £ x Q= such that rel= o 72 = relinj 
(equality is a consequence of minimality; see Proposition 5.2). We make the follow- 
ing observations: 

(1) As rel| o1Z = rel inj , TZ C {(a,q) \ q G rell(a)}. 

(2) For each a G Qs, 72(a) ^ (otherwise (a, a) will be present in relj n j but not in 
rel|o^). 

(3) qsTZqi (72 is a simulation). 

(4) For each a € Qs, each fi G <5(a) and each a TZ q there exists a // G <5(q) such 
that /x /x' (72 is a simulation). 

(5) For any relation 72. i C {{a,q) \ q G rel 7 (a)} such that 72 C 72i, fi a G 5(a), 
fi q G S(q) we have that /x a <n implies that fi a diHi ^q- 

Now, the first observation above implies that in the algorithm in Figure 8 initially 
72 is contained within the relation denoted by the variable R. TZ is also contained 
in the relation denoted by the variable R id, the first time the variable R id takes 
a non-empty value. From this point on, we claim 72 is always contained in the 
relations R Q id and R. This claim is a consequence of the fourth and the fifth 
observations which ensure that every time R is updated, 72 is contained in the 
relation denoted by R. Finally note that second and third observations ensure that 
the algorithm can never declare the counterexample to be invalid and hence by 
termination (Proposition 5.3), the algorithm must return "valid". □ 

A careful analysis of the special structure of the validating simulation yields 
better bounds than that reported in [Baier et al. 2000] for general simulation. 

Theorem 5.6. Let M. be an MDP, = an equivalence relation compatible with 
A4, and ips a safety property such that M.= \f ips- Let (£, relinj) be a counterexample 
for M.= and ips generated using Theorem 3.8. Then the algorithm in Figure 8 
returns "valid" iff (£, relinj) is valid and consistent with (M,=). Let um and hum 
be the number of vertices and edges, respectively, in the underlying labeled graph 
Gt(M). The algorithm shown in Figure 8 runs in time 0(n 2 M m 2 M ). 

Proof. Thanks to Propositions 5.4 and 5.5, the result will follow if we show 
that the running time of the algorithm is 0(n 2 M m 2 M ). 

Observe that the outermost while loop runs for as long as R changes. If for 
each a G Qs, we define s a = |rell(a)| = |rell o relj n j (a) | , a bound on the number of 
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iterations of the outermost loop is X^aeQ £ s a = n M, a s each state of Q belongs to at 
most one relX(a). Next let us define d a to be number of outgoing edges from the set 
relX(a) and d q to be the out-degree of q in Gt(M). Clearly for each state a £ Qs, 
the total number of tests of the form /i ^R old £*' is bounded by X^ereii(a) d a d q = d\. 
Thus, the total number of tests \i <r oU m' m a single iteration of the outermost 
loop is bounded by X^aeQ£ d a — m M- Now, because each state of Q belongs to at 
most one rell o rel; n j(a), the test \x <R old \i' simply requires one to check that for 
each b £ Qf, ix(6) < J2 q -(a q)eR old ^'(l) ano ^ tnus can be done in 0(um) time. Thus 
we don't need to compute flows in bipartite networks, as is required in the general 
case [Baier et al. 2000]. The total running time is, therefore, 0(n 2 M m 2 M ). □ 

We make some observations about the validity checking algorithm. 

(1) For linear time properties and non-probabilistic systems, checking validity of a 
counterexample simply determines if the first state in the counterexample can 
be simulated by the initial state of the system by going backwards from the last 
state of the counterexample. The same idea can be exploited for probabilistic 
systems as well if the underlying unlabeled graph of the counterexample £ is 
a tree (or more generally a DAG); the resulting algorithm will depend on the 
height of the counterexample and cut the running time of the algorithm by a 
factor of n_M- 

(2) Theorem 3.4 observes that for weak safety formulas, counterexamples whose 
underlying graph is a tree can be found by "unrolling" the minimal MDP 
counterexample. Instead of first explicitly unrolling the counterexample and 
then checking, one can unroll the counterexample "on the fly" while checking 
validity. This algorithm is presented in Section 5.3, after our discussion on 
refinement so as to not interrupt the flow. The crucial idea is to decide when to 
stop unrolling which is made by keeping track of the satisfaction of subformulas 
at various states. The running time of the algorithm will be 0(h ■ umi^m), 
where h is the height of the unrolled tree. Thus depending on um and h, one 
could either compute the actual simulation relation, or simply check whether 
the tree of height h is simulated. 

(3) One can construct the graph of maximal strongly connected components of 
G (,{£), and compute the simulation relation on each maximal strongly connect 
component, in the order of their topological sort. While this new algorithm 
will not yield better asymptotic bounds, it may work better in practice. 

To complete the description of the CEGAR approach, all we need to do is describe 
the refinement step. However, before we proceed, we describe a result and give 
some notations for the case when the counterexample generated by the algorithm 
in Figure 4 is declared by the counterexample checking algorithm to be invalid. 

PROPOSITION 5.7. If the algorithm in Figure 8 returns ("invalid", a, [i,R u, 
R), then for all q G R id(d) \ TZ(d) and all \i\ £ S(q), fi ^R old Hi- 

Proof. Immediate consequence of the algorithm. □ 

Notation: If the algorithm in Figure 8 returns ("invalid", a, n,R id, R) then 

— a is said to be an invalidating abstract state; 
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{90,91,93 




{97,98} 




{99} 




{910} 



{911} 



Fig. 9. The abstraction K' ab 

— [i € Sq £ (a) is said to be an invalidating transition; and 

— the pair {R id, R) is said to be the invalidating witness for a and /i. 

5.2 Refining Abstractions 

The last step in the abstraction-refinement loop is to refine the abstraction in the 
case when the counterexample is invalid. The algorithm in Figure 8, concludes the 
invalidity of the counterexample, when it finds some abstract state a such that a 
is a state of the counterexample and a is not simulated by any concrete state in 
rel 7 (a), or when a is the initial state of the counterexample and it is not simulated 
by the initial state of Ai. At this point, we will refine the abstraction by refining 
the equivalence = that was used to construct the abstract model in the first place. 
The goal of the refinement step is for it to be "counterexample guided" . The ideal 
situation is one where the spurious counterexample is "eliminated" by the refine- 
ment step. However, as we remind the reader, this is not achieved in the CEGAR 
approach for non-probabilistic systems. We, therefore, begin (Section 5.2.1) by 
motivating and defining the notion of a "good refinement". We show that good 
refinements do indeed lead to progress in the CEGAR approach. After this, in 
Section 5.2.2, we present a refinement algorithm along with a proof that it results 
in good refinements. 

5.2.f Good Refinements. We begin by recalling the refinement step in the CE- 
GAR approach for non-probabilistic systems through an example, to demonstrate 
that refinement does not lead to the elimination of the counterexample. The ex- 
ample, however, motivates what the refinement step does indeed achieve, leading 
us to the notion of good refinements. 

Example 5.8. As in Example 5.1, consider the Kripke structure /Cex from Ex- 
ample 4.1 along with the abstraction /C a b (these structures are also given in Fig- 
ures 5 and 6). The LTL safety-property <fi — n(-iP) is violated by /C a b and the 
counterexample generation algorithm may generate the spurious counterexample 
£cex 3 = {<?o, qi, 93} -> {55, qa) {97, q%\ -> {<Zii}- Now, the counterexample check- 
ing algorithm in [Clarke et al. 2000; Clarke et al. 2002] starts from the last state of 
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the counterexample and proceeds backwards, checking at each point whether any 
of the concrete states corresponding to the abstract state in the counterexample 
can exhibit the counterexample from that point onwards. The algorithm finds that 
there is a path q§ — > qj — > qn in /C ex which simulates {95,96} — ► {97,9s} — > {911} 
but finds that there is no transition from {qo, 91, 93} to q§. At this point, it declares 
the counterexample to be invalid. The refinement step then breaks the equivalence 
class post({go, 9i, 93}) = {95,96} into {q^} and {95}. The resulting abstraction K' ab 
shown in Figure 9 still has the "spurious" counterexample IC' cexs = {90,91,93} - » 
{95} —> {97,9s} -» {?ii} "contained" within /C ceX3 . 

Though, in Example 5.8, the counterexample is not eliminated by the refinement, 
progress is nonetheless made. Considering the example carefully, one notes that 
breaking {q 5 , q 6 } could have yielded two possible new paths - tC' ceX3 = {qo, 9i, #3} — * 
{95} -> {97,9s} -» {911} and /C" eX3 = {90, 91,93} -> {9e} -> {97,9s} -> {qn}- How- 
ever, only one path IC' ceX3 is a simulated by K! ab while the other path /C" eX3 is n °t 
simulated by KI ab and hence has been "eliminated". Thus, what is eliminated is at 
least one simulation relation that is "contained" in the original spurious counterex- 
ample. We capture this concept for MDPs as follows. 

Definition: Let Ai be an MDP with states Q, ~ and = be equivalence relations on 
Q compatible with M such that ~ C=, and -0s be a safety property. Let (8, TZ) be a 
counterexample for A4= and tps, where Qg is the set of states of 8 with initial state 
qg. Finally let 9~ by the initial state of A4~. We say that ~ is a good =-refinement 
for (£, K) if there is some R'CQ £ x such that (q £ ,q~) e W , re\Z = o W = K 
but TV is not a canonical simulation (of £ by _M~). If no such TV exists, we say 
that ~ is a bad =-refinement for (£,TV). 

Intuitively, a good =-refinement ~ ensures that (£,TV) is not a counterexample 
for and tps- Observe that the conditions on TV ensure that TV is one of 

the possible proofs that M. violates tps "contained" within the counterexample 
{8, TV). This presents yet another justification for formally treating the simulation 
relation (or proof) as part of the notion of a counterexample. In the absence of the 
simulation relation, it is difficult justify why refinement is "counterexample guided" 
given that the behavior (i.e., 8) itself may not be eliminated. 

Remark: Before presenting the consequences of good refinements, we would like 
examine the formal definition more carefully, in order to highlight the subtle reasons 
why all the points in the definition are needed. 

(1) Observe that TV satisfying rel^ = o TV = TZ always exist: take TV to be any 
relation such that for each qo £ Qf, TV {qo) = U[ q }= e -K(q )X[ q ]= where ^[ g ]= is 
any non-empty subset of {[9i]~ £ Q~ | [9i]~ C [9] = }. Further, any TV such that 
re '~ = °TV = TZ must be of this form. 

(2) We demand rel^ _ o TV = TZ rather than rel" = o TV C TZ. One can easily come 
up with TZ' which satisfies rel^ = o TZ' C TZ but is not a simulation (take, e.g. 
TV = 0). Note also that if (8, TZ) is a minimal counterexample then the set 
7^(90) is non-empty for each q E Qg. Thus, by taking ~ to be = and taking 
X[ q ]^ to be empty for some [q}= above, we will ensure that the resulting TV is 
not a simulation and rel^ _ o TZ' C TZ. Hence we would have declared = to be 
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a good —refinement if we had not required the equality! Thus we may not be 
able to guarantee any progress in the CEGAR loop (see Proposition 5.10). 

(3) We require that {qe,q~) 6 R'. If we had not required this condition then 
one could have achieved a "good" refinement by just breaking the initial state 
and taking TZ' to be any relation such that re\Z = o TZ' = TZ but fails to be 
a simulation just because TZ' relates qs to equivalence classes that does not 
contain the initial state of M.~. 

Suppose (£,1Z) is a counterexample for M= and ips', hence £ \f ips- Therefore, 
if TZ' C Q £ x Q~ is a canonical simulation then (£,7Z') is a counterexample for 
A4~ and ips- The following proposition says that if re\Z = o1Z' = TZ and (£,1Z) 
is invalid for (_M,=), then (£,TZ') is invalid for (M,~). Thus, a good refinement 
ensures that at least one of counterexamples "contained" within (£,1Z) is not a 
counterexample, and thereby eliminates at least one spurious counterexample that 
would not be eliminated by a bad refinement. 

PROPOSITION 5.9. Let M be a MDP with Q as the set of states, ~ and = be 
equivalence relations compatible with M such that ~C=. Let (£,TZ) be a coun- 
terexample for M= and ips with Qs as the set of states. Let TZ' C Q £ x Q~ be a 
canonical simulation such that re\Z = o TZ' = TZ. Then [£,TZ') is a counterexample 
for _M~ and ips- Further, if(£,TZ) is invalid for (A4,=) then (£,TZ') is invalid for 

Proof. That (£ , 72.') is a a counterexample for M.~ and ips follows from the fact 
that 1Z' is a simulation and that £ \f ips- Assume, by way of contradiction, that 
(£, 1Z') is valid and consistent with (M., ~). Then there exists a canonical simulation 
TZ C Q E x Q such that re\Z o K C TZ! . This implies that (rel^ _ o relS,) o TZ a C 
re '~ = 72'- But the left hand side is rel^ o 72o while the right hand side is TZ. This 
implies that (£, TZ) is a valid and consistent with (M, =) (with TZo as the validating 
simulation). A contradiction! □ 

We conclude this section by showing that good refinements ensure progress in 
the CEGAR loop. 

Proposition 5.10. Let M. be an MDP, and ~ and = be equivalence relations 
compatible with M. such that ~C=. Let {£,TZ) be a counterexample for A4= and 
ips- If — is a good = -refinement for (£,TZ), then ~ C=. 

Proof. Fix a relation TZ' C x Q such that rel^ _ o TZ' = TZ but TZ' is not a 
canonical simulation. We now proceed by contradiction. Assume that c±—=. Then 
M.= and M.~ are the same abstract MDP and rel" = is the identity relation. Thus 
TZ' and TZ are the same relation. But TZ' is not a simulation which contradicts that 
fact that (£,TZ) is a counterexample for M= and ips- □ 

5.2.2 Algorithm for Refinement. In this section, we will show how an abstract 
model can be refined based on a spurious counterexample obtained as in Theorem 
3.8. Before presenting our algorithm, we recall how the refinement step proceeds 
for non-probabilistic systems, through an example. This will help us highlight a 
couple of key points about the refinement step in the non-probabilistic case. 
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Example 5.11. Recall in the (non-probabilistic) Example 5.8, the counterex- 
ample checking step proceeded from the last state of the counterexample trace 
£cex 3 = {90,91,93} — » {95,96} — ► {97,98} — > {911} and confirmed that the con- 
crete state 96 can simulate the path {95,96} — ► {97,9s} — > {911}- Since there is 
no "concrete" transition from {90,91,93} to q$, the counterexample checking step 
concludes that the counterexample is invalid. The state {90,91,93} is the coun- 
terpart of "invalidating abstract state", the transition t = {90,91,93} — ► {95,96} 
is the counterpart of "invaliding transition", {q 6 } is counterpart of R id({q5, 96}) 
and is counterpart of R({qo, 91, 93}). The refinement step for non-probabilistic 
case is obtained by splitting the equivalence class post(i, {90, 91, 93}) = {95, 96} into 
{9e} = Roid({q5, 9e}) and {q 5 } = {q 5 , q 6 } \ R i d ({q 5 , q 6 }). 

On the other hand, suppose for the Kripke structure K, ex and its abstraction /C a t>, 
the counterexample IC ceX2 = {90,91,93} — * {94} — ► {910} —> {911} is chosen instead 
of /C ce x 3 . In this case the counterexample generation algorithm declares the coun- 
terexample to be invalid when the initial state of IC ex , 90, fails to be in R({qo, qi, 93}) 
during the counterexample checking algorithm. In this case, {90,91,93} is the "in- 
validating" abstract state; {90,92,93} is the counterpart of -R o w({90, 9i, 93}) and 
{93} is the counterpart of R({qo, 91, 93})- For this case, the invalidating abstract 
state {90,91,93} is itself broken into R old ({q ,qi,q 3 }) \ R{{qo, 9i, 93}) = {90, 9i} 
and {go, 9i, 93} \ {RoidiUo, 9i, 93}) \ R{{qo, 9i, 93}) = {93})- Thus, in this case the 
invalidating abstract state itself is broken into equivalence classes and not its suc- 
cessor! 

Let us examine the refinement step outlined in Example 5.11 more carefully. 
There are two cases to consider: when the invalidating abstract state is not the 
initial state, where we only split abstract state that is the target of the invalidat- 
ing transition; and when the invalidating abstract state is the initial state of the 
counterexample, where we also split the invalidating abstract state. 

To generalize to probabilistic systems, we make the following observations. Sup- 
pose M= is the abstraction of M with respect to =, and let (£, reLj) be a coun- 
terexample for A4= and ips obtained as in Theorem 3.8 and which is invalid for 
(M,=). If a is the invalidating abstract state, and \x £ 8s(a) is the invalidating 
transition, then post(/x, a) may contain several states (including a itself). Hence, 
our refinement step will be forced to split several equivalence classes instead of 
one as in the case of non-probabilistic systems. Next, we observe that in the case 
when the counterexample is a "path" (or more generally a DAG), the algorithm 
to check validity only needs to "process" each state of the counterexample once. 
Hence, if (R id,R) is the invalidating witness, then R id(a) = rell(a). Therefore, 
Roid(a) \R(a) is always re\~(a) except (possibly) when the invalidating state a is 
the initial state of the counterexample. This is the primary reason why for non- 
probabilistic systems the invalidating abstract state is never split, except when it is 
the initial state. However, when analyzing counterexamples that could be general 
MDPs, the counterexample checking algorithm will need to "process" each state 
multiple times, and then R u(a>) need not be rell(a), at the time the counterexam- 
ple is deemed to be invalid. Thus, in our refinement algorithm, we will be forced 
to also split the invalidating abstract state. 
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The above intuitions are formalized in the refinement step shown in Figure 10; 
recall that for each d € Qs, we have R(d) C R id{d) C rel 7 (d). We conclude by 
showing that the resulting refinement is a good refinement (and hence progress is 
ensured in the CEGAR loop). 

Theorem 5.12. Let (£, re I i n j ) be a counterexample, generated using Theorem 3.8, 
for M= and safety property ips, where M= is the abstraction of M with respect to 
the compatible equivalence relation =. If the counterexample checking algorithm in 
Figure 8 returns ("invalid", a, p,R id, R), then the refinement ~C= obtained as 
in Figure 10 is a good =-refinement for {£, relmj). 

Proof. Let £ = (Qs,qs,5s , Lf). Recall that = (Q~, 6~, L~) is the 
abstract MDP for A4 and ~, where the set of states of A4~ are equivalence classes 
under ~. Consider the functional relation 1Z C Qs x Q~ defined as follows. 

— (a, a') G 1Z iff a' is the ^-equivalence class R id(d) \ R(a). 

— For each b G post(/i, a) \ d, (b, b') G 1Z iff b' is the ^-equivalence class Roidib). 

— For each c <E Q £ \ (post(^, a) U a), (c, c') G 1Z iff = a 

Please note that it is easy to see that by construction (qs,q~) G 7£; if a is not the 
initial state then the observation follows immediately and otherwise, observe that 
<Zj G R id{a) \ R(d). Further, we clearly have relS, = oTZ = rel; n j. We shall show 
that 1Z is not a canonical simulation and hence we can conclude that ~ is a good 
refinement. 

Now, let ao G Q~ be the ^-equivalence class 1Z id{a)\R{d) . Observe that (a, ao) G 
1Z. Next, recall that the violating transition p G 5s(d). Hence the desired result 
will follow if we can show that for each G <5~(ao) we have that p -^tz po- 

We proceed by contradiction. Let pJ be such that p! G <5~(a ) and p, -<ti p! . By 
definition of abstractions, there is a q G rell(a ) = R id{a) \ R{a) and G (5(g) 
such that // = From p! ', we can conclude the following. 

—Ma) < Vi(Roid(a) \R(a)) < m(R id(a)). 

— For each 6 G post(/x, a) \ a, /i(6) < pi(Roid(b)). 

—For c G Q £ \ (post(^i, a) U a), p(c) = < pi(R id(c)). 

For each d G Qf, it follows from construction that R id(d) C rell(rf). Therefore for 
all d~0)di G Qs such that do 7^ d~i, Roid(do) H R id(d~i) = 0. It now follows easily 
from the above observations that ^n old f-i which contradicts Proposition 5.7. □ 

5.3 Counter-example checking for weak safety 

In this section we outline an algorithm that given a counterexample £ for an MDP 
A4 and weak safety property ipws either determines that £ is not a valid counterex- 
ample, or finds a finite unrolling of £ that is simulated by M, and witnesses the 
fact that M. does not satisfy ipws- The algorithm unrolls £ on the fly, and does not 
construct the unrolled MDP explicitly. The running time of the algorithm depends 
on the height of the unrolling, which if small, can result in faster checking than the 
algorithm shown in Figure 8. Before presenting the algorithm, we introduce some 
notation that we will find useful in describing the algorithm. Recall that given an 
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The refinement ~ is obtained from the equivalence = as follows. 

If a is the invalidating abstract state, fi 6 (a) the invalidating transition 

and (R id, R) the invalidating witness then: 

The =-cquivalcncc class reU, (a) is broken into 

new ~-cquivalcnce classes R id(&) \ R(a) and rel 7 (a) \ (R id{a) \ R(a)) 
For each b G post(/i, a) \ a, the =-equivalence class rel 7 (6) is broken into 

new ^-equivalence classes R id(b) and rel 7 (fe) \ R id(b) 
No other =-equivalence class is refined. 



Fig. 10. Refinement algorithm based on invalid counterexamples 

MDP M, a state q of M and k G N the fc-th unrolling of MDP rooted at q is 
denoted by M.\. 

Notation: Given k G N and states q, q' G M, we say that q <k q' if M\ < M\ . 
Given a PCTL formula ip we say that q Ihfc ip if A4 q k lh ip. 

We observe the following two facts. If q ^ q' then q < k q' for all k. The 
proof of Theorem 3.4 implies that for a weak safety formula ipws if <Z V~ "4>ws, 
then there is a fco s.t. g l^fc tpws- These two facts can be combined to obtain a 
counter-example checking algorithm for the weak-safety fragment of PCTL as we 
shall describe shortly. 

For the rest of the Section, we fix the following notation. M = (Q, qx, 5, L) is 
the (original) MDP that we are checking against weak safety property ipws an d 
= is an equivalence relation that is compatible with M. Assuming M= violates 
the safety property ^wSi w e w hl denote the minimal counterexample obtained as 
in Theorem 3.8 by (£,rel in j). Let £ = (Q £ ,q £ ,S £ ,L £ ) . For a state a = [q]= in 
abstraction M = 1 rell(a) = {q'\q = q'} is the concretization map. The relation 
{(a,q) | q G relX(a),a G Q £ } shall be denoted by Rj. Finally, we shall use ipsL 
to denote the strict liveness formula obtained by negating tpws- SLSubForm^si) 
will denote the set of PCTL strict liveness subformulas of V'sl and Path Form (iPsl) 
will denote the set of path subformulas of ipsL 4 - 

The proposed algorithm iteratively constructs the relations Rk = Ri fl <k and 
Satfe = {(a, ip) | a Ihfc ip,a G Qs 7 ip G SLSubForm(7/;5L)}- We make the following 
observations. 

(1) If the set Rk(a) = {q | (a,q) G Rk} becomes for some k and a G Q £ , then 
(£ , reUj) is invalid for (_M,=). We can also call the counterexample invalid if 
the initial concrete state qx is not contained in Rk{qe)- 

(2) If Satfe(<7£, V'sl) and qx G Rk(qs) then qx lr-fc ipsL also. Thus, the concrete 
MDP violates the given safety property and we can report this. 

This iteration must end as a consequence of Theorem 3.4. The computation also 
needs to compute the function MaxProbfe defined as follows. Given a G Q £ and a 
path formula <\> G PathForm^s'rJ, MaxProbfe gives the maximum probability (over 
all schedulers) of <j> being true in The relations Rk+i, Sat^+i and MaxProbfc+i 



4 A path subformula of a PCTL-liveness formula arc formulas of the kind XipL an d 
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can be computed using Rk, Satfe, and MaxProb^ and do not need other previous 
values. 



Initially 

-Rcurr = {(a,q) | q G rel 7 (a),a G Qs} 

Sat curr = {(a, ip) a I ho V 1 : V 7 € SI_SubForm(t/> SL ), a € Qf} 
MaxProb curr [a][(/)] = MaxProbo(a)((/>) for a G Qg,0 G Path Form (i/isl) 
While (true) 
do 

If Ql £ Rcurr (qs) return "Counter-example not simulated" 
If Sat(q £ ,iPsl) return "Safety Violated" 
for each a £ Qg 
do 

Rtmp.a = {q | (a,q) g _R C urr and V/i £ 5f (a)3/i' G 5(g). /u ^H curr //} 
If -Rtmp.a = return "Counter-example not simulated" 
COMPUTER, Sat curr , MaxProb CU rr, Sattmp.a, MaxProb t mp,s) 

od 

Satcurr = {(a,1p) I V G Sattmp.a} 
Rcurr = {{a, q) | q G .Rtmp.a} 

MaxProb curr [a][c/>] = MaxP 

robtmp,a t° r each a G G)£,(/> G Path Form (i/j) 

od 

The procedure COMPUTE returns Sattmp.a: the set of sub-formulas of tpsL satisfied by a 
in the "next" unrolling of the tree with root a. It also returns MaxProbtmp.a, 
that given a path formula <j> gives the maximum probability of <j> being true in 
the next unrolling. COMPUTE is defined as follows. 

COMPUTER, Satcurr, MaxProbcurr, Sat tmp ,a, MaxProbtmp.a) 

Fix an enumeration ipi, ... , tp n of the set {ip \ ip G PathForm(i/>,5x)U 

SLSubForm(^gi)} such that size((fi) < size(ipj) for i < j. 
Initially 
Sattmp.a — 

MaxProb tmPl a[</>] = for all <j> G Path Form (tpsh)- 
For i = 1 to n 

If ipi is p and p G Lf(a) then Sattmp.a = Sattmp.a U {p} 

If tpi is -ip and p ^ Lf(a) then Sattmp.a = Sattmp.a U {^p} 

If ipi is Vi V V>2 and (i/>i G Sattmp.a or tp2 G Sattmp.a) then 

Sattmp.a — Sattmp.a U {ipi} 

If ipi is A V>2, ij>i G Sattmp.a and i/>2 G Sattmp.a then 

Sattmp.a — Sattmp.a U {ipi} 

If ipi is -iV< p (<t>) and MaxProbtmp.a [</>] > V 
then Sattmp.a = Sattmp.a U {ipi) 

If is Xt/> then 

MaxProbtmp,a[^i] = max Mgi£(a) fi({b | (5,i/>) G Sat CU rr}) 
If ipi is Vl W 4>2 then 

If i/>2 G Sattmp.a then MaxProbtmp.a [ipi] = 1 

else 

If Vl ^ Sattmp.a then MaxProbtmp.a [v>i] = 

else MaxProbtmp.a [<£i] = max Mgi£(a) S oeQ£ (M 6 )) (MaxProbcurrlV"! W ^H&D 



Fig. 11. On the fly algorithm for checking Strict Liveness 
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Figure 11 gives the details of this algorithm. At the beginning of (k + l)-th 
unrolling of the while loop, the relations i? C urr and Sat curr are the relations Rk 
and Satfc respectively. The (doubly-indexed) array MaxProb curr [a] [ip] is the func- 
tion MaxProbfc[a][V>] ■ Within the while loop, i?tmp,a is the set Rk+i(a) while 
Sat tmp ,a is the set {ip \ a \\~k+i ip € SLSubForm^sL)}, and MaxProb t mp,a[V'] is 
MaxProbfc + i[a][i/>]. R C un, Satcurr, and MaxProb CU rr are updated after i?tm P ,a, Sat tmp ,a 
and MaxProb tmPia are computed for all states a £ Qg. The following proposition 
follows easily from the observations made in the Section. 

Proposition 5.13. The algorithm in Figure 11 terminates. If the algorithm 
returns "Counter-example not simulated" then the counterexample obtained using 
Theorem 3.8 is not valid. If the algorithm returns "Safety Violated" then M. IK 
ipws- 

Finally we observe that the algorithm in Figure 11 may be made more efficient 
in practice as follows. First, since we are dealing with strict liveness fragment, 
the sequence Satfc is an increasing sequence and the function MaxProbfc[a][i/>] < 
MaxProbfc + i[a][i/>]. Hence, only needs to compute Satfc + i \ Satfc and MaxProbfc + i — 
MaxProbfc. This optimization shall be explored in future work. 

6. RELATED WORK 

Abstraction Schemes: Abstractions have been extensively studied in the con- 
text of probabilistic systems. General issues in defining good abstractions as well 
as specific proposals for families of abstract models are presented in [Jonsson and 
Larsen 1991; Huth 2004; Norman 2004; Huth 2005; D'Argenio et al. 2001; 2002; 
Fecher et al. 2006; Katoen et al. 2007; Monniaux 2005; Kwiatkowska et al. 2006; 
Mclver and Morgan 2004]. Recently, theorem-prover based algorithms for con- 
structing abstractions of probabilistic systems based on predicates have been pre- 
sented [Wachter et al. 2007]. Another notion that has been recently proposed is 
the notion of a "magnifying- lens abstraction" [de Alfaro and Roy 2007], which can 
be used to assist in the model checking process, by approximating the measure of 
the satisfaction of path formulas for sets of concrete states; the method is not an 
abstraction in the traditional sense in that neither is an abstract model explicitly 
constructed, nor is the model used for reasoning, one that simulates the concrete 
model. 

Counterexamples: The notion of counterexamples is critical for the approach of 
counterexample guided abstraction refinement. Criteria for defining counterexam- 
ples are identified in [Clarke et al. 2002], along with a notion of counterexamples 
for branching-time properties and non-probabilistic systems. The problem of defin- 
ing counterexamples for probabilistic systems has received considerable attention 
recently. Starting from the seminal papers [Aljazzar et al. 2005; Han and Katoen 
2007a], the notion of sets of paths with high measure as counterexamples has been 
used for DTMCs, CTMCs, and MDPs [Han and Katoen 2007a; 2007b; Aljazzar 
and Leue 2007] . Another definition that has been proposed is that of DTMCs (or 
purely probabilistic models) in [Chatterjee et al. 2005; Hermanns et al. 2008]. Our 
notion of counterexample is different from these proposals and we demonstrate that 
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these other proposals are not rich enough for the class of properties we consider. 

Automatic Abstraction-Refinement: In the context of probabilistic systems, 
automatic abstract-refinement was first considered in [D'Argenio ct al. 2001; 2002]. 
There are two main differences with our work. First, they consider only reachability 
properties. Second, the refinement process outlined in [DArgenio et al. 2001; 2002] 
is not counterexample based, but rather based on partition refinement. What this 
means is that their refinement is biased towards separating states that are not 
bisimilar, rather than states that are "distinguished" by the property, and so it is 
likely that their method refines more than needed. 

Counterexample guided refinement has been used as the basis of synthesizing win- 
ning strategies for 2-player stochastic games in [Chatterjee et al. 2005]. Though the 
problem 2-player games is more general than verification, the specific model consid- 
ered by [Chatterjee et al. 2005] has some peculiarities and so does not subsume the 
problem or its solution presented here. First, in their model, states are partitioned 
into "non-deterministic" states that have purely nondeterministic transitions and 
"probabilistic" states that have purely probabilistic transitions. The abstraction 
does not abstract any of the "probabilistic states"; only the "non-deterministic" 
states are collapsed. This results in larger abstract models, and obviates certain 
issues in counterexample checking that we deal with. Second, they take counterex- 
amples to be finite models without nondetcrminism. They can do this because they 
consider a simpler class of properties than we do, and as we show in Section 3.3, 
DTMCs are not rich enough for all of safe-PCTL. Next, their counterexample check- 
ing algorithm is different than even the one used in the context of non-probabilistic 
systems. They consider a counterexample to be valid only if all the concrete states 
corresponding to the abstract states in the counterexample can simulate the be- 
havior captured by the counterexample. Thus, they deem certain counterexamples 
to be spurious, even if they will be recognized as providing enough evidence for 
the violation of the property by other CEGAR schemes (including ours). Finally, 
they do not have a precise statement characterizing the qualities of their refinement 
algorithm. 

In a recent paper, [Hermanns et al. 2008] consider CEGAR for probabilistic 
systems. They consider very special types of reachability properties namely, those 
that can be expressed by formulas of the form V< p (tpi U ^2) where ^1 and tp2 are 
propositions (or boolean combinations of propositions). For this class of properties, 
they use DTMCs as the notion of counterexamples : the counterexample obtained 
is a pair (S, M s ) where S is a memoryless scheduler. As we show, DTMCs cannot 
serve as counterexample for the richer class of properties considered here. For the 
counterexample checking algorithm, they generate a finite set, ap, of abstract paths 
of A4f in decreasing order of measures such that the total measure of these paths 
is > p. Then, they build (on-the-fly) a "concrete" scheduler which maximizes the 
measure of the paths in ap that are simulated by the original MDP. Let p to tai be 
the maximum probability (under all schedulers) of ipi U tp2 being satisfied by the 
abstract MDP, p ap be the total measure of ap and p max be the maximum measure 
of "abstract" paths in ap simulated by the concrete MDP. If p max > p then the 
counterexample is declared to be valid and if p tota | — p ap + p max < p then the 
counterexample is declared to be invalid and the abstraction refined. If neither is 
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the case, then [Hermanns et al. 2008] heuristically decide either to generate more 
abstract paths or to refine the abstraction. The refinement is based upon refining 
some "spurious" abstract path (namely, a path that is not simulated by the concrete 
system). There is, however, no formal statement characterizing progress based on 
the refinement algorithm outlined in [Hermanns et al. 2008]. 

7. CONCLUSIONS AND FUTURE WORK 

We presented a CEGAR framework for MDPs, where an MDP M is abstracted 
by another MDP A defined using an equivalence on the states of M.. Our main 
contributions when presenting this framework were a definition for the notion of 
a counterexample, along with algorithms to compute counterexamples, check their 
validity and perform automatic refinement based on an invalid counterexample. 

There are a number of interesting questions left open for future investigation. 
First these ideas need to be implemented and experimented with. In order for this 
approach to be scalable, symbolic algorithms for a lot of the steps outlined here, 
will be required. Next, when constructing minimal counterexamples, the order 
in which transitions are considered for elimination, crucially affects the final size 
of the counterexample. Good heuristics for ordering transitions to obtain small 
counterexamples, must be identified. 
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